2023-07-20T18:25:58.009 INFO:root:teuthology version: 0.0.1.dev153+g79491cb 2023-07-20T18:25:58.010 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} --archive /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400 --owner scheduled_pdonnell@teuthology --name pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi -- /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/orig.config.yaml 2023-07-20T18:25:58.076 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T18:25:58.176 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400 branch: wip-pdonnell-testing-20230719.201220 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} email: pdonnell@redhat.com first_in_suite: false job_id: '7345400' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230719.201220 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client dirsize rbytes: false client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync rbytes: false syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230719.201220 sha1: 78296bf9274b016ce68ff036026e51a56af22f7c owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8388 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 sleep_before_teardown: 0 subset: 12/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230719.201220 suite_path: /home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 78296bf9274b016ce68ff036026e51a56af22f7c syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi088.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa distributed 1 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/norstats teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 79491cb69dd37a1eafb93b3471998174bdfa2fd9 timestamp: 2023-07-20_16:47:59 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-20T18:25:58.176 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa; will attempt to use it 2023-07-20T18:25:58.177 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks 2023-07-20T18:25:58.177 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-20T18:25:58.179 INFO:teuthology.task.internal:Checking packages... 2023-07-20T18:25:58.207 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '1b0b2eb005a7c51d97a788b9900d511e1394b537' 2023-07-20T18:25:58.207 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-20T18:25:58.208 INFO:teuthology.packaging:ref: None 2023-07-20T18:25:58.208 INFO:teuthology.packaging:tag: None 2023-07-20T18:25:58.208 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230719.201220 2023-07-20T18:25:58.209 INFO:teuthology.packaging:sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:25:58.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230719.201220 2023-07-20T18:25:58.559 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5040.g1b0b2eb0 2023-07-20T18:25:58.559 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-20T18:25:58.566 INFO:teuthology.task.internal:no buildpackages task found 2023-07-20T18:25:58.566 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-20T18:25:58.575 INFO:teuthology.task.internal:Saving configuration 2023-07-20T18:25:58.595 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-20T18:25:58.601 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T18:25:58.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:20:24.630419', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T18:25:58.654 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:20:24.629175', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T18:25:58.679 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:20:24.627195', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T18:25:58.679 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-20T18:25:58.689 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-20T18:25:58.690 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-20T18:25:58.690 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-20T18:25:58.690 INFO:teuthology.run_tasks:Running task console_log... 2023-07-20T18:25:58.810 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f27c68d73a0>, signals=[15]) 2023-07-20T18:25:58.812 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-20T18:25:58.821 INFO:teuthology.task.internal:Opening connections... 2023-07-20T18:25:58.821 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2023-07-20T18:25:58.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:25:58.900 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2023-07-20T18:25:58.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:25:58.968 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2023-07-20T18:25:58.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:25:59.036 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-20T18:25:59.045 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2023-07-20T18:25:59.068 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2023-07-20T18:25:59.069 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2023-07-20T18:25:59.127 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T18:25:59.128 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T18:25:59.128 INFO:teuthology.orchestra.run.smithi057.stdout:ID="rhel" 2023-07-20T18:25:59.128 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="fedora" 2023-07-20T18:25:59.128 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="8.6" 2023-07-20T18:25:59.128 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el8" 2023-07-20T18:25:59.128 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T18:25:59.129 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31" 2023-07-20T18:25:59.129 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T18:25:59.129 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T18:25:59.129 INFO:teuthology.orchestra.run.smithi057.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T18:25:59.129 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T18:25:59.129 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:25:59.130 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T18:25:59.130 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T18:25:59.130 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T18:25:59.130 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T18:25:59.131 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2023-07-20T18:25:59.155 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2023-07-20T18:25:59.176 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2023-07-20T18:25:59.177 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2023-07-20T18:25:59.232 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T18:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T18:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout:ID="rhel" 2023-07-20T18:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="fedora" 2023-07-20T18:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8.6" 2023-07-20T18:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8" 2023-07-20T18:25:59.233 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T18:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2023-07-20T18:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T18:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T18:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T18:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T18:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:25:59.234 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T18:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T18:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T18:25:59.235 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T18:25:59.235 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2023-07-20T18:25:59.255 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2023-07-20T18:25:59.275 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2023-07-20T18:25:59.276 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2023-07-20T18:25:59.334 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T18:25:59.335 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T18:25:59.335 INFO:teuthology.orchestra.run.smithi192.stdout:ID="rhel" 2023-07-20T18:25:59.335 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="fedora" 2023-07-20T18:25:59.335 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8.6" 2023-07-20T18:25:59.335 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8" 2023-07-20T18:25:59.335 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T18:25:59.335 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2023-07-20T18:25:59.336 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T18:25:59.336 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T18:25:59.336 INFO:teuthology.orchestra.run.smithi192.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T18:25:59.336 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T18:25:59.336 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:25:59.336 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T18:25:59.337 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T18:25:59.337 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T18:25:59.337 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T18:25:59.337 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2023-07-20T18:25:59.363 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-20T18:25:59.378 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-20T18:25:59.386 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-20T18:25:59.387 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2023-07-20T18:25:59.390 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2023-07-20T18:25:59.393 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2023-07-20T18:25:59.412 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-20T18:25:59.418 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-20T18:25:59.419 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2023-07-20T18:25:59.450 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2023-07-20T18:25:59.453 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2023-07-20T18:25:59.541 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-20T18:25:59.626 INFO:teuthology.run_tasks:Running task kernel... 2023-07-20T18:25:59.649 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-07-20T18:25:59.650 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-07-20T18:25:59.650 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-07-20T18:25:59.650 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T18:25:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:25:59.671 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-07-20T18:25:59.980 DEBUG:teuthology.task.kernel:sha1 for client.0 is c685da6f56cc77f2cfb9c4221be3960522616caf 2023-07-20T18:25:59.981 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "c685da6f56cc77f2cfb9c4221be3960522616caf"... 2023-07-20T18:25:59.981 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2023-07-20T18:26:00.001 INFO:teuthology.orchestra.run.smithi057.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-20T18:26:00.002 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs c685da6f56cc77f2cfb9c4221be3960522616caf 2023-07-20T18:26:00.002 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-07-20T18:26:00.003 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum upgrade -y linux-firmware 2023-07-20T18:26:01.208 INFO:teuthology.orchestra.run.smithi057.stdout:Updating Subscription Management repositories. 2023-07-20T18:26:01.209 INFO:teuthology.orchestra.run.smithi057.stdout:Unable to read consumer identity 2023-07-20T18:26:02.661 INFO:teuthology.orchestra.run.smithi057.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-07-20T18:26:05.228 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 16 MB 00:02 2023-07-20T18:26:08.903 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 404 kB/s | 24 kB 00:00 2023-07-20T18:26:09.679 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-07-20T18:26:09.680 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-07-20T18:26:09.681 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-07-20T18:26:09.801 INFO:teuthology.task.kernel:Downloading kernel c685da6f56cc77f2cfb9c4221be3960522616caf on client.0... 2023-07-20T18:26:09.801 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c685da6f56cc77f2cfb9c4221be3960522616caf 2023-07-20T18:26:10.064 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/c685da6f56cc77f2cfb9c4221be3960522616caf/centos/8/flavors/default/x86_64/ 2023-07-20T18:26:10.064 DEBUG:teuthology.orchestra.run.smithi057:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.5.0_rc1_gc685da6f56cc-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/c685da6f56cc77f2cfb9c4221be3960522616caf/centos/8/flavors/default/x86_64/ --input-file=- 2023-07-20T18:26:10.067 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi057.front.sepia.ceph.com to complete... 2023-07-20T18:27:56.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20 18:27:56 URL:https://4.chacra.ceph.com/r/kernel/testing/c685da6f56cc77f2cfb9c4221be3960522616caf/centos/8/flavors/default/x86_64/kernel-6.5.0_rc1_gc685da6f56cc-1.x86_64.rpm [380683892/380683892] -> "/tmp/kernel.x86_64.rpm" [1] 2023-07-20T18:27:56.285 INFO:teuthology.orchestra.run.smithi057.stderr:FINISHED --2023-07-20 18:27:56-- 2023-07-20T18:27:56.285 INFO:teuthology.orchestra.run.smithi057.stderr:Total wall clock time: 1m 46s 2023-07-20T18:27:56.285 INFO:teuthology.orchestra.run.smithi057.stderr:Downloaded: 1 files, 363M in 1m 46s (3.43 MB/s) 2023-07-20T18:27:56.287 INFO:teuthology.task.kernel:Installing kernel c685da6f56cc77f2cfb9c4221be3960522616caf on client.0... 2023-07-20T18:27:56.287 DEBUG:teuthology.orchestra.run.smithi057:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-07-20T18:27:57.968 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying... ######################################## 2023-07-20T18:27:58.027 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing... ######################################## 2023-07-20T18:27:58.120 INFO:teuthology.orchestra.run.smithi057.stdout:Updating / installing... 2023-07-20T18:28:38.579 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-6.5.0_rc1_gc685da6f56cc-1 ######################################## 2023-07-20T18:29:05.679 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-07-20T18:29:05.679 DEBUG:teuthology.orchestra.run.smithi057:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-07-20T18:29:05.732 INFO:teuthology.orchestra.run.smithi057.stdout:/boot/System.map-6.5.0-rc1-gc685da6f56cc 2023-07-20T18:29:05.733 INFO:teuthology.orchestra.run.smithi057.stdout:/boot/config-6.5.0-rc1-gc685da6f56cc 2023-07-20T18:29:05.733 INFO:teuthology.orchestra.run.smithi057.stdout:/boot/vmlinuz-6.5.0-rc1-gc685da6f56cc 2023-07-20T18:29:05.733 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc 2023-07-20T18:29:05.733 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel 2023-07-20T18:29:05.733 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch 2023-07-20T18:29:05.733 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86 2023-07-20T18:29:05.734 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto 2023-07-20T18:29:05.734 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-07-20T18:29:05.734 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-07-20T18:29:05.734 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-07-20T18:29:05.734 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-07-20T18:29:05.734 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-07-20T18:29:05.734 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-07-20T18:29:05.734 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-07-20T18:29:05.735 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/crc32c-intel.ko 2023-07-20T18:29:05.735 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-07-20T18:29:05.735 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-07-20T18:29:05.735 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-07-20T18:29:05.735 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-07-20T18:29:05.735 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/serpent-avx2.ko 2023-07-20T18:29:05.735 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-07-20T18:29:05.736 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-07-20T18:29:05.736 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-07-20T18:29:05.736 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-07-20T18:29:05.736 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-07-20T18:29:05.736 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-07-20T18:29:05.736 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-07-20T18:29:05.736 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/kernel 2023-07-20T18:29:05.736 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/kernel/cpu 2023-07-20T18:29:05.737 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/kernel/cpu/mce 2023-07-20T18:29:05.737 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-07-20T18:29:05.737 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/kvm 2023-07-20T18:29:05.737 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/kvm/kvm-amd.ko 2023-07-20T18:29:05.737 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/kvm/kvm-intel.ko 2023-07-20T18:29:05.737 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/arch/x86/kvm/kvm.ko 2023-07-20T18:29:05.737 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto 2023-07-20T18:29:05.738 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/ansi_cprng.ko 2023-07-20T18:29:05.738 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/anubis.ko 2023-07-20T18:29:05.738 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/arc4.ko 2023-07-20T18:29:05.738 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/async_tx 2023-07-20T18:29:05.738 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/async_tx/async_memcpy.ko 2023-07-20T18:29:05.738 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/async_tx/async_pq.ko 2023-07-20T18:29:05.738 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/async_tx/async_raid6_recov.ko 2023-07-20T18:29:05.738 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/async_tx/async_tx.ko 2023-07-20T18:29:05.739 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/async_tx/async_xor.ko 2023-07-20T18:29:05.739 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/async_tx/raid6test.ko 2023-07-20T18:29:05.739 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/authenc.ko 2023-07-20T18:29:05.739 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/authencesn.ko 2023-07-20T18:29:05.739 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/blake2b_generic.ko 2023-07-20T18:29:05.739 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/blowfish_common.ko 2023-07-20T18:29:05.739 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/blowfish_generic.ko 2023-07-20T18:29:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/camellia_generic.ko 2023-07-20T18:29:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/cast5_generic.ko 2023-07-20T18:29:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/cast6_generic.ko 2023-07-20T18:29:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/cast_common.ko 2023-07-20T18:29:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/ccm.ko 2023-07-20T18:29:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/cmac.ko 2023-07-20T18:29:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/crc32_generic.ko 2023-07-20T18:29:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/crypto_engine.ko 2023-07-20T18:29:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/crypto_user.ko 2023-07-20T18:29:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/cts.ko 2023-07-20T18:29:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/des_generic.ko 2023-07-20T18:29:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/dh_generic.ko 2023-07-20T18:29:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/echainiv.ko 2023-07-20T18:29:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/essiv.ko 2023-07-20T18:29:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/fcrypt.ko 2023-07-20T18:29:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/khazad.ko 2023-07-20T18:29:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/lz4.ko 2023-07-20T18:29:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/lz4hc.ko 2023-07-20T18:29:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/md4.ko 2023-07-20T18:29:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/michael_mic.ko 2023-07-20T18:29:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/pcbc.ko 2023-07-20T18:29:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/pcrypt.ko 2023-07-20T18:29:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/rmd160.ko 2023-07-20T18:29:05.743 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/seed.ko 2023-07-20T18:29:05.743 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/serpent_generic.ko 2023-07-20T18:29:05.743 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/tcrypt.ko 2023-07-20T18:29:05.743 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/tea.ko 2023-07-20T18:29:05.743 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/twofish_common.ko 2023-07-20T18:29:05.743 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/twofish_generic.ko 2023-07-20T18:29:05.743 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/vmac.ko 2023-07-20T18:29:05.743 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/wp512.ko 2023-07-20T18:29:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/xcbc.ko 2023-07-20T18:29:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/xor.ko 2023-07-20T18:29:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/crypto/xxhash_generic.ko 2023-07-20T18:29:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers 2023-07-20T18:29:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/acpi 2023-07-20T18:29:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/acpi/acpi_ipmi.ko 2023-07-20T18:29:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/acpi/acpi_pad.ko 2023-07-20T18:29:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/acpi/custom_method.ko 2023-07-20T18:29:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/acpi/ec_sys.ko 2023-07-20T18:29:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/acpi/platform_profile.ko 2023-07-20T18:29:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/acpi/sbs.ko 2023-07-20T18:29:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/acpi/sbshc.ko 2023-07-20T18:29:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/acpi/video.ko 2023-07-20T18:29:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata 2023-07-20T18:29:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/acard-ahci.ko 2023-07-20T18:29:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/ahci_platform.ko 2023-07-20T18:29:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/ata_generic.ko 2023-07-20T18:29:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/libahci_platform.ko 2023-07-20T18:29:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_acpi.ko 2023-07-20T18:29:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_ali.ko 2023-07-20T18:29:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_amd.ko 2023-07-20T18:29:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_artop.ko 2023-07-20T18:29:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_atiixp.ko 2023-07-20T18:29:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_atp867x.ko 2023-07-20T18:29:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_cmd640.ko 2023-07-20T18:29:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_cmd64x.ko 2023-07-20T18:29:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_cypress.ko 2023-07-20T18:29:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_efar.ko 2023-07-20T18:29:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_hpt366.ko 2023-07-20T18:29:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_hpt37x.ko 2023-07-20T18:29:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_hpt3x2n.ko 2023-07-20T18:29:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_hpt3x3.ko 2023-07-20T18:29:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_it8213.ko 2023-07-20T18:29:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_it821x.ko 2023-07-20T18:29:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_jmicron.ko 2023-07-20T18:29:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_marvell.ko 2023-07-20T18:29:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_mpiix.ko 2023-07-20T18:29:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_netcell.ko 2023-07-20T18:29:05.749 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_ninja32.ko 2023-07-20T18:29:05.749 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_ns87410.ko 2023-07-20T18:29:05.749 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_ns87415.ko 2023-07-20T18:29:05.749 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_oldpiix.ko 2023-07-20T18:29:05.749 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_opti.ko 2023-07-20T18:29:05.749 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_optidma.ko 2023-07-20T18:29:05.749 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_pcmcia.ko 2023-07-20T18:29:05.749 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_pdc2027x.ko 2023-07-20T18:29:05.750 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-07-20T18:29:05.750 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_piccolo.ko 2023-07-20T18:29:05.750 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_rdc.ko 2023-07-20T18:29:05.750 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_sch.ko 2023-07-20T18:29:05.750 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_serverworks.ko 2023-07-20T18:29:05.750 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_sil680.ko 2023-07-20T18:29:05.750 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_sis.ko 2023-07-20T18:29:05.751 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_sl82c105.ko 2023-07-20T18:29:05.751 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_triflex.ko 2023-07-20T18:29:05.751 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pata_via.ko 2023-07-20T18:29:05.751 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/pdc_adma.ko 2023-07-20T18:29:05.751 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_inic162x.ko 2023-07-20T18:29:05.751 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_mv.ko 2023-07-20T18:29:05.751 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_nv.ko 2023-07-20T18:29:05.751 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_promise.ko 2023-07-20T18:29:05.752 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_qstor.ko 2023-07-20T18:29:05.752 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_sil.ko 2023-07-20T18:29:05.752 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_sil24.ko 2023-07-20T18:29:05.752 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_sis.ko 2023-07-20T18:29:05.752 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_svw.ko 2023-07-20T18:29:05.752 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_sx4.ko 2023-07-20T18:29:05.752 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_uli.ko 2023-07-20T18:29:05.753 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_via.ko 2023-07-20T18:29:05.753 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ata/sata_vsc.ko 2023-07-20T18:29:05.753 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/auxdisplay 2023-07-20T18:29:05.753 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/auxdisplay/ks0108.ko 2023-07-20T18:29:05.753 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/base 2023-07-20T18:29:05.753 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/base/regmap 2023-07-20T18:29:05.753 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/base/regmap/regmap-i2c.ko 2023-07-20T18:29:05.753 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/base/regmap/regmap-mmio.ko 2023-07-20T18:29:05.754 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/bcma 2023-07-20T18:29:05.754 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/bcma/bcma.ko 2023-07-20T18:29:05.754 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block 2023-07-20T18:29:05.754 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/aoe 2023-07-20T18:29:05.754 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/aoe/aoe.ko 2023-07-20T18:29:05.754 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/brd.ko 2023-07-20T18:29:05.754 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/drbd 2023-07-20T18:29:05.754 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/drbd/drbd.ko 2023-07-20T18:29:05.755 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/floppy.ko 2023-07-20T18:29:05.755 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/loop.ko 2023-07-20T18:29:05.755 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/mtip32xx 2023-07-20T18:29:05.755 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-07-20T18:29:05.755 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/nbd.ko 2023-07-20T18:29:05.755 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/null_blk 2023-07-20T18:29:05.755 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/null_blk/null_blk.ko 2023-07-20T18:29:05.755 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/pktcdvd.ko 2023-07-20T18:29:05.756 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/rbd.ko 2023-07-20T18:29:05.756 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/virtio_blk.ko 2023-07-20T18:29:05.756 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/xen-blkback 2023-07-20T18:29:05.756 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-07-20T18:29:05.756 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/xen-blkfront.ko 2023-07-20T18:29:05.756 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/zram 2023-07-20T18:29:05.756 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/block/zram/zram.ko 2023-07-20T18:29:05.757 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char 2023-07-20T18:29:05.757 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/hangcheck-timer.ko 2023-07-20T18:29:05.757 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/hw_random 2023-07-20T18:29:05.757 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/hw_random/amd-rng.ko 2023-07-20T18:29:05.757 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/hw_random/intel-rng.ko 2023-07-20T18:29:05.757 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-07-20T18:29:05.757 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/hw_random/via-rng.ko 2023-07-20T18:29:05.757 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/hw_random/virtio-rng.ko 2023-07-20T18:29:05.758 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/ipmi 2023-07-20T18:29:05.758 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-07-20T18:29:05.758 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-07-20T18:29:05.758 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-07-20T18:29:05.758 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/ipmi/ipmi_si.ko 2023-07-20T18:29:05.758 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-07-20T18:29:05.758 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-07-20T18:29:05.758 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/lp.ko 2023-07-20T18:29:05.759 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/mwave 2023-07-20T18:29:05.759 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/mwave/mwave.ko 2023-07-20T18:29:05.759 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/ppdev.ko 2023-07-20T18:29:05.759 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/tlclk.ko 2023-07-20T18:29:05.759 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/tpm 2023-07-20T18:29:05.759 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/tpm/tpm.ko 2023-07-20T18:29:05.759 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/tpm/tpm_atmel.ko 2023-07-20T18:29:05.760 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/tpm/tpm_infineon.ko 2023-07-20T18:29:05.760 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/tpm/tpm_nsc.ko 2023-07-20T18:29:05.760 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/tpm/tpm_tis.ko 2023-07-20T18:29:05.760 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-07-20T18:29:05.760 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/uv_mmtimer.ko 2023-07-20T18:29:05.760 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/char/virtio_console.ko 2023-07-20T18:29:05.760 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/cpufreq 2023-07-20T18:29:05.760 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-07-20T18:29:05.761 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-07-20T18:29:05.761 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/cpufreq/p4-clockmod.ko 2023-07-20T18:29:05.761 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-07-20T18:29:05.761 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/cpufreq/powernow-k8.ko 2023-07-20T18:29:05.761 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/cpufreq/speedstep-lib.ko 2023-07-20T18:29:05.761 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto 2023-07-20T18:29:05.761 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/ccp 2023-07-20T18:29:05.761 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-07-20T18:29:05.762 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/ccp/ccp.ko 2023-07-20T18:29:05.762 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/intel 2023-07-20T18:29:05.762 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/intel/qat 2023-07-20T18:29:05.762 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/intel/qat/qat_common 2023-07-20T18:29:05.762 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-07-20T18:29:05.762 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-07-20T18:29:05.762 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-07-20T18:29:05.763 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/padlock-aes.ko 2023-07-20T18:29:05.763 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/padlock-sha.ko 2023-07-20T18:29:05.763 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/virtio 2023-07-20T18:29:05.763 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-07-20T18:29:05.763 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/dca 2023-07-20T18:29:05.763 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/dca/dca.ko 2023-07-20T18:29:05.763 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/devfreq 2023-07-20T18:29:05.763 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-07-20T18:29:05.764 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/dma 2023-07-20T18:29:05.764 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/dma/dw 2023-07-20T18:29:05.764 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/dma/dw/dw_dmac.ko 2023-07-20T18:29:05.764 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/dma/ioat 2023-07-20T18:29:05.764 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/dma/ioat/ioatdma.ko 2023-07-20T18:29:05.764 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac 2023-07-20T18:29:05.764 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/amd64_edac.ko 2023-07-20T18:29:05.765 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/e752x_edac.ko 2023-07-20T18:29:05.765 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/edac_mce_amd.ko 2023-07-20T18:29:05.765 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/i3000_edac.ko 2023-07-20T18:29:05.765 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/i3200_edac.ko 2023-07-20T18:29:05.765 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/i5100_edac.ko 2023-07-20T18:29:05.765 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/i5400_edac.ko 2023-07-20T18:29:05.765 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/i7300_edac.ko 2023-07-20T18:29:05.765 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/i7core_edac.ko 2023-07-20T18:29:05.766 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/i82975x_edac.ko 2023-07-20T18:29:05.766 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/ie31200_edac.ko 2023-07-20T18:29:05.766 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/sb_edac.ko 2023-07-20T18:29:05.766 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/edac/x38_edac.ko 2023-07-20T18:29:05.766 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/firmware 2023-07-20T18:29:05.766 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/firmware/edd.ko 2023-07-20T18:29:05.766 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/firmware/iscsi_ibft.ko 2023-07-20T18:29:05.766 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/gpio 2023-07-20T18:29:05.767 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/gpio/gpio-ich.ko 2023-07-20T18:29:05.767 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/gpio/gpio-viperboard.ko 2023-07-20T18:29:05.767 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid 2023-07-20T18:29:05.767 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-appleir.ko 2023-07-20T18:29:05.767 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-aureal.ko 2023-07-20T18:29:05.767 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-axff.ko 2023-07-20T18:29:05.767 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-dr.ko 2023-07-20T18:29:05.768 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-elecom.ko 2023-07-20T18:29:05.768 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-elo.ko 2023-07-20T18:29:05.768 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-emsff.ko 2023-07-20T18:29:05.768 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-gaff.ko 2023-07-20T18:29:05.768 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-gt683r.ko 2023-07-20T18:29:05.768 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-gyration.ko 2023-07-20T18:29:05.768 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-holtek-kbd.ko 2023-07-20T18:29:05.768 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-holtek-mouse.ko 2023-07-20T18:29:05.769 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-holtekff.ko 2023-07-20T18:29:05.769 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-hyperv.ko 2023-07-20T18:29:05.769 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-icade.ko 2023-07-20T18:29:05.769 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-keytouch.ko 2023-07-20T18:29:05.769 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-kye.ko 2023-07-20T18:29:05.769 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-lcpower.ko 2023-07-20T18:29:05.769 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-led.ko 2023-07-20T18:29:05.769 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-lenovo.ko 2023-07-20T18:29:05.770 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-logitech-dj.ko 2023-07-20T18:29:05.770 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-07-20T18:29:05.770 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-multitouch.ko 2023-07-20T18:29:05.770 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-ortek.ko 2023-07-20T18:29:05.770 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-penmount.ko 2023-07-20T18:29:05.770 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-petalynx.ko 2023-07-20T18:29:05.770 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-picolcd.ko 2023-07-20T18:29:05.771 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-pl.ko 2023-07-20T18:29:05.771 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-plantronics.ko 2023-07-20T18:29:05.771 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-primax.ko 2023-07-20T18:29:05.771 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-rmi.ko 2023-07-20T18:29:05.771 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-arvo.ko 2023-07-20T18:29:05.771 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-common.ko 2023-07-20T18:29:05.771 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-isku.ko 2023-07-20T18:29:05.771 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-kone.ko 2023-07-20T18:29:05.772 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-07-20T18:29:05.772 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-konepure.ko 2023-07-20T18:29:05.772 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-07-20T18:29:05.772 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-lua.ko 2023-07-20T18:29:05.772 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-pyra.ko 2023-07-20T18:29:05.772 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-ryos.ko 2023-07-20T18:29:05.772 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat-savu.ko 2023-07-20T18:29:05.772 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-roccat.ko 2023-07-20T18:29:05.773 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-saitek.ko 2023-07-20T18:29:05.773 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-samsung.ko 2023-07-20T18:29:05.773 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-sensor-hub.ko 2023-07-20T18:29:05.773 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-sjoy.ko 2023-07-20T18:29:05.773 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-sony.ko 2023-07-20T18:29:05.773 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-speedlink.ko 2023-07-20T18:29:05.773 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-steelseries.ko 2023-07-20T18:29:05.774 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-sunplus.ko 2023-07-20T18:29:05.774 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-thrustmaster.ko 2023-07-20T18:29:05.774 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-tivo.ko 2023-07-20T18:29:05.774 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-tmff.ko 2023-07-20T18:29:05.774 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-topseed.ko 2023-07-20T18:29:05.774 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-twinhan.ko 2023-07-20T18:29:05.774 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-uclogic.ko 2023-07-20T18:29:05.774 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-waltop.ko 2023-07-20T18:29:05.775 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-wiimote.ko 2023-07-20T18:29:05.775 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-xinmo.ko 2023-07-20T18:29:05.775 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-zpff.ko 2023-07-20T18:29:05.775 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/hid-zydacron.ko 2023-07-20T18:29:05.775 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/uhid.ko 2023-07-20T18:29:05.775 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hid/wacom.ko 2023-07-20T18:29:05.775 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hv 2023-07-20T18:29:05.775 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hv/hv_balloon.ko 2023-07-20T18:29:05.776 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hv/hv_utils.ko 2023-07-20T18:29:05.776 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hv/hv_vmbus.ko 2023-07-20T18:29:05.776 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon 2023-07-20T18:29:05.776 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/abituguru.ko 2023-07-20T18:29:05.776 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/abituguru3.ko 2023-07-20T18:29:05.776 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/acpi_power_meter.ko 2023-07-20T18:29:05.776 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ad7414.ko 2023-07-20T18:29:05.776 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ad7418.ko 2023-07-20T18:29:05.777 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adc128d818.ko 2023-07-20T18:29:05.777 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adm1025.ko 2023-07-20T18:29:05.777 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adm1026.ko 2023-07-20T18:29:05.777 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adm1029.ko 2023-07-20T18:29:05.777 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adm1031.ko 2023-07-20T18:29:05.777 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adm9240.ko 2023-07-20T18:29:05.777 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ads7828.ko 2023-07-20T18:29:05.778 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adt7410.ko 2023-07-20T18:29:05.778 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adt7411.ko 2023-07-20T18:29:05.778 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adt7462.ko 2023-07-20T18:29:05.778 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adt7470.ko 2023-07-20T18:29:05.778 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adt7475.ko 2023-07-20T18:29:05.778 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/adt7x10.ko 2023-07-20T18:29:05.778 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/amc6821.ko 2023-07-20T18:29:05.778 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/applesmc.ko 2023-07-20T18:29:05.779 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/asb100.ko 2023-07-20T18:29:05.779 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/asc7621.ko 2023-07-20T18:29:05.779 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/asus_atk0110.ko 2023-07-20T18:29:05.779 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/atxp1.ko 2023-07-20T18:29:05.779 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/coretemp.ko 2023-07-20T18:29:05.779 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-07-20T18:29:05.779 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/dme1737.ko 2023-07-20T18:29:05.780 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ds1621.ko 2023-07-20T18:29:05.780 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ds620.ko 2023-07-20T18:29:05.780 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/emc1403.ko 2023-07-20T18:29:05.780 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/emc6w201.ko 2023-07-20T18:29:05.780 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/f71805f.ko 2023-07-20T18:29:05.780 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/f71882fg.ko 2023-07-20T18:29:05.780 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/f75375s.ko 2023-07-20T18:29:05.780 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/fam15h_power.ko 2023-07-20T18:29:05.781 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/fschmd.ko 2023-07-20T18:29:05.781 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/g760a.ko 2023-07-20T18:29:05.781 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/g762.ko 2023-07-20T18:29:05.781 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/gl518sm.ko 2023-07-20T18:29:05.781 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/gl520sm.ko 2023-07-20T18:29:05.781 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/hwmon-vid.ko 2023-07-20T18:29:05.781 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/i5500_temp.ko 2023-07-20T18:29:05.781 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/i5k_amb.ko 2023-07-20T18:29:05.782 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ibmaem.ko 2023-07-20T18:29:05.782 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ibmpex.ko 2023-07-20T18:29:05.782 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ina209.ko 2023-07-20T18:29:05.782 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ina2xx.ko 2023-07-20T18:29:05.782 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/it87.ko 2023-07-20T18:29:05.782 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/k10temp.ko 2023-07-20T18:29:05.782 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/k8temp.ko 2023-07-20T18:29:05.782 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lineage-pem.ko 2023-07-20T18:29:05.783 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm63.ko 2023-07-20T18:29:05.783 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm73.ko 2023-07-20T18:29:05.783 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm75.ko 2023-07-20T18:29:05.783 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm77.ko 2023-07-20T18:29:05.783 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm78.ko 2023-07-20T18:29:05.783 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm80.ko 2023-07-20T18:29:05.783 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm83.ko 2023-07-20T18:29:05.784 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm85.ko 2023-07-20T18:29:05.784 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm87.ko 2023-07-20T18:29:05.784 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm90.ko 2023-07-20T18:29:05.784 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm92.ko 2023-07-20T18:29:05.784 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm93.ko 2023-07-20T18:29:05.784 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm95234.ko 2023-07-20T18:29:05.784 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm95241.ko 2023-07-20T18:29:05.784 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/lm95245.ko 2023-07-20T18:29:05.785 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ltc2945.ko 2023-07-20T18:29:05.785 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ltc4151.ko 2023-07-20T18:29:05.785 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ltc4215.ko 2023-07-20T18:29:05.785 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ltc4222.ko 2023-07-20T18:29:05.785 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ltc4245.ko 2023-07-20T18:29:05.785 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ltc4260.ko 2023-07-20T18:29:05.785 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/ltc4261.ko 2023-07-20T18:29:05.785 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/max16065.ko 2023-07-20T18:29:05.786 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/max1619.ko 2023-07-20T18:29:05.786 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/max1668.ko 2023-07-20T18:29:05.786 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/max197.ko 2023-07-20T18:29:05.786 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/max6639.ko 2023-07-20T18:29:05.786 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/max6650.ko 2023-07-20T18:29:05.786 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/max6697.ko 2023-07-20T18:29:05.786 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/mcp3021.ko 2023-07-20T18:29:05.786 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/nct6683.ko 2023-07-20T18:29:05.787 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/nct6775-core.ko 2023-07-20T18:29:05.787 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/nct6775.ko 2023-07-20T18:29:05.787 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/nct7802.ko 2023-07-20T18:29:05.787 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pc87360.ko 2023-07-20T18:29:05.787 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pc87427.ko 2023-07-20T18:29:05.787 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pcf8591.ko 2023-07-20T18:29:05.787 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus 2023-07-20T18:29:05.787 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-07-20T18:29:05.788 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-07-20T18:29:05.788 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-07-20T18:29:05.788 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/max16064.ko 2023-07-20T18:29:05.788 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/max34440.ko 2023-07-20T18:29:05.788 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/max8688.ko 2023-07-20T18:29:05.788 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-07-20T18:29:05.788 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-07-20T18:29:05.789 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-07-20T18:29:05.789 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-07-20T18:29:05.789 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-07-20T18:29:05.789 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-07-20T18:29:05.789 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/powr1220.ko 2023-07-20T18:29:05.789 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/sch5627.ko 2023-07-20T18:29:05.789 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/sch5636.ko 2023-07-20T18:29:05.789 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/sch56xx-common.ko 2023-07-20T18:29:05.790 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/sht15.ko 2023-07-20T18:29:05.790 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/sht21.ko 2023-07-20T18:29:05.790 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/shtc1.ko 2023-07-20T18:29:05.790 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/sis5595.ko 2023-07-20T18:29:05.790 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/smsc47b397.ko 2023-07-20T18:29:05.790 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/smsc47m1.ko 2023-07-20T18:29:05.790 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/smsc47m192.ko 2023-07-20T18:29:05.790 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/thmc50.ko 2023-07-20T18:29:05.791 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/tmp102.ko 2023-07-20T18:29:05.791 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/tmp103.ko 2023-07-20T18:29:05.791 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/tmp401.ko 2023-07-20T18:29:05.791 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/tmp421.ko 2023-07-20T18:29:05.791 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/via-cputemp.ko 2023-07-20T18:29:05.791 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/via686a.ko 2023-07-20T18:29:05.791 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/vt1211.ko 2023-07-20T18:29:05.791 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/vt8231.ko 2023-07-20T18:29:05.792 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/w83627ehf.ko 2023-07-20T18:29:05.792 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/w83627hf.ko 2023-07-20T18:29:05.792 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/w83781d.ko 2023-07-20T18:29:05.792 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/w83791d.ko 2023-07-20T18:29:05.792 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/w83792d.ko 2023-07-20T18:29:05.792 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/w83793.ko 2023-07-20T18:29:05.792 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/w83795.ko 2023-07-20T18:29:05.793 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/w83l785ts.ko 2023-07-20T18:29:05.793 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/hwmon/w83l786ng.ko 2023-07-20T18:29:05.793 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c 2023-07-20T18:29:05.793 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/algos 2023-07-20T18:29:05.793 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-07-20T18:29:05.793 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-07-20T18:29:05.793 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses 2023-07-20T18:29:05.793 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-07-20T18:29:05.794 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-07-20T18:29:05.794 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-07-20T18:29:05.794 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-07-20T18:29:05.794 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-07-20T18:29:05.794 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-07-20T18:29:05.794 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-07-20T18:29:05.794 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-07-20T18:29:05.794 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-i801.ko 2023-07-20T18:29:05.795 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-isch.ko 2023-07-20T18:29:05.795 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-07-20T18:29:05.795 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-07-20T18:29:05.795 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-07-20T18:29:05.795 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-parport.ko 2023-07-20T18:29:05.795 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-07-20T18:29:05.795 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-07-20T18:29:05.795 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-07-20T18:29:05.796 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-07-20T18:29:05.796 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-07-20T18:29:05.796 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-07-20T18:29:05.796 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-via.ko 2023-07-20T18:29:05.796 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-07-20T18:29:05.796 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-07-20T18:29:05.796 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/i2c-dev.ko 2023-07-20T18:29:05.797 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/i2c-mux.ko 2023-07-20T18:29:05.797 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-07-20T18:29:05.797 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/i2c-smbus.ko 2023-07-20T18:29:05.797 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/i2c/i2c-stub.ko 2023-07-20T18:29:05.797 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband 2023-07-20T18:29:05.797 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/core 2023-07-20T18:29:05.797 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/core/ib_cm.ko 2023-07-20T18:29:05.797 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/core/ib_core.ko 2023-07-20T18:29:05.798 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/core/ib_umad.ko 2023-07-20T18:29:05.798 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-07-20T18:29:05.798 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/core/iw_cm.ko 2023-07-20T18:29:05.798 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/core/rdma_cm.ko 2023-07-20T18:29:05.798 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-07-20T18:29:05.799 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/hw 2023-07-20T18:29:05.799 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/hw/cxgb4 2023-07-20T18:29:05.799 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-07-20T18:29:05.799 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/hw/mlx4 2023-07-20T18:29:05.799 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-07-20T18:29:05.799 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/hw/mlx5 2023-07-20T18:29:05.799 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-07-20T18:29:05.799 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/hw/mthca 2023-07-20T18:29:05.800 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-07-20T18:29:05.800 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp 2023-07-20T18:29:05.800 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp/ipoib 2023-07-20T18:29:05.800 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-07-20T18:29:05.800 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp/iser 2023-07-20T18:29:05.800 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-07-20T18:29:05.800 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp/isert 2023-07-20T18:29:05.800 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-07-20T18:29:05.801 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp/srp 2023-07-20T18:29:05.801 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-07-20T18:29:05.801 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp/srpt 2023-07-20T18:29:05.801 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-07-20T18:29:05.801 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input 2023-07-20T18:29:05.801 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/gameport 2023-07-20T18:29:05.801 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-07-20T18:29:05.802 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/gameport/fm801-gp.ko 2023-07-20T18:29:05.802 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/gameport/gameport.ko 2023-07-20T18:29:05.802 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/gameport/lightning.ko 2023-07-20T18:29:05.802 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/gameport/ns558.ko 2023-07-20T18:29:05.802 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/joydev.ko 2023-07-20T18:29:05.802 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/keyboard 2023-07-20T18:29:05.802 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/keyboard/gpio_keys.ko 2023-07-20T18:29:05.802 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/mouse 2023-07-20T18:29:05.803 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/mouse/appletouch.ko 2023-07-20T18:29:05.803 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/mouse/bcm5974.ko 2023-07-20T18:29:05.803 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/mouse/cyapatp.ko 2023-07-20T18:29:05.803 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/mouse/elan_i2c.ko 2023-07-20T18:29:05.803 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/mouse/sermouse.ko 2023-07-20T18:29:05.803 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-07-20T18:29:05.803 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/mouse/synaptics_usb.ko 2023-07-20T18:29:05.803 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/mouse/vsxxxaa.ko 2023-07-20T18:29:05.804 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/rmi4 2023-07-20T18:29:05.804 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/rmi4/rmi_core.ko 2023-07-20T18:29:05.804 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/serio 2023-07-20T18:29:05.804 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/serio/altera_ps2.ko 2023-07-20T18:29:05.804 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/serio/arc_ps2.ko 2023-07-20T18:29:05.804 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-07-20T18:29:05.804 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/serio/serio_raw.ko 2023-07-20T18:29:05.805 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/input/sparse-keymap.ko 2023-07-20T18:29:05.805 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/iommu 2023-07-20T18:29:05.805 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/iommu/amd 2023-07-20T18:29:05.805 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/iommu/amd/iommu_v2.ko 2023-07-20T18:29:05.805 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds 2023-07-20T18:29:05.805 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/leds-blinkm.ko 2023-07-20T18:29:05.805 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/leds-lm3530.ko 2023-07-20T18:29:05.805 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/leds-lp3944.ko 2023-07-20T18:29:05.806 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/leds-ss4200.ko 2023-07-20T18:29:05.806 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/trigger 2023-07-20T18:29:05.806 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-07-20T18:29:05.806 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-07-20T18:29:05.806 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-07-20T18:29:05.806 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-07-20T18:29:05.806 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-07-20T18:29:05.806 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-07-20T18:29:05.807 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-07-20T18:29:05.807 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-07-20T18:29:05.807 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md 2023-07-20T18:29:05.807 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/bcache 2023-07-20T18:29:05.807 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/bcache/bcache.ko 2023-07-20T18:29:05.807 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-bio-prison.ko 2023-07-20T18:29:05.807 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-cache-smq.ko 2023-07-20T18:29:05.807 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-cache.ko 2023-07-20T18:29:05.808 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-crypt.ko 2023-07-20T18:29:05.808 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-delay.ko 2023-07-20T18:29:05.808 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-flakey.ko 2023-07-20T18:29:05.808 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-log-userspace.ko 2023-07-20T18:29:05.808 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-multipath.ko 2023-07-20T18:29:05.808 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-queue-length.ko 2023-07-20T18:29:05.808 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-raid.ko 2023-07-20T18:29:05.809 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-round-robin.ko 2023-07-20T18:29:05.809 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-service-time.ko 2023-07-20T18:29:05.809 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-switch.ko 2023-07-20T18:29:05.809 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-thin-pool.ko 2023-07-20T18:29:05.809 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/dm-verity.ko 2023-07-20T18:29:05.809 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/faulty.ko 2023-07-20T18:29:05.809 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/linear.ko 2023-07-20T18:29:05.809 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/multipath.ko 2023-07-20T18:29:05.810 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/persistent-data 2023-07-20T18:29:05.810 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-07-20T18:29:05.810 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/raid0.ko 2023-07-20T18:29:05.810 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/raid1.ko 2023-07-20T18:29:05.810 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/raid10.ko 2023-07-20T18:29:05.810 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/md/raid456.ko 2023-07-20T18:29:05.810 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/message 2023-07-20T18:29:05.810 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/message/fusion 2023-07-20T18:29:05.811 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/message/fusion/mptbase.ko 2023-07-20T18:29:05.811 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/message/fusion/mptctl.ko 2023-07-20T18:29:05.811 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/message/fusion/mptfc.ko 2023-07-20T18:29:05.811 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/message/fusion/mptlan.ko 2023-07-20T18:29:05.811 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/message/fusion/mptsas.ko 2023-07-20T18:29:05.811 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/message/fusion/mptscsih.ko 2023-07-20T18:29:05.811 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/message/fusion/mptspi.ko 2023-07-20T18:29:05.812 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/mfd 2023-07-20T18:29:05.812 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/mfd/lpc_ich.ko 2023-07-20T18:29:05.812 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/mfd/lpc_sch.ko 2023-07-20T18:29:05.812 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/mfd/mfd-core.ko 2023-07-20T18:29:05.812 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/mfd/sm501.ko 2023-07-20T18:29:05.812 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/mfd/viperboard.ko 2023-07-20T18:29:05.812 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/mfd/vx855.ko 2023-07-20T18:29:05.812 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/mfd/wl1273-core.ko 2023-07-20T18:29:05.813 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc 2023-07-20T18:29:05.813 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/altera-stapl 2023-07-20T18:29:05.813 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-07-20T18:29:05.813 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/apds9802als.ko 2023-07-20T18:29:05.813 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/apds990x.ko 2023-07-20T18:29:05.813 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/bh1770glc.ko 2023-07-20T18:29:05.813 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/cb710 2023-07-20T18:29:05.813 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/cb710/cb710.ko 2023-07-20T18:29:05.814 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/eeprom 2023-07-20T18:29:05.814 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/eeprom/at24.ko 2023-07-20T18:29:05.814 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/eeprom/eeprom.ko 2023-07-20T18:29:05.814 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-07-20T18:29:05.814 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/eeprom/max6875.ko 2023-07-20T18:29:05.814 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/enclosure.ko 2023-07-20T18:29:05.814 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/hpilo.ko 2023-07-20T18:29:05.814 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/isl29003.ko 2023-07-20T18:29:05.815 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/isl29020.ko 2023-07-20T18:29:05.815 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/lis3lv02d 2023-07-20T18:29:05.815 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-07-20T18:29:05.815 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-07-20T18:29:05.815 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/mei 2023-07-20T18:29:05.815 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/mei/mei-me.ko 2023-07-20T18:29:05.815 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/mei/mei-txe.ko 2023-07-20T18:29:05.816 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/mei/mei.ko 2023-07-20T18:29:05.816 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/sgi-gru 2023-07-20T18:29:05.816 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/sgi-gru/gru.ko 2023-07-20T18:29:05.816 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/sgi-xp 2023-07-20T18:29:05.816 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/sgi-xp/xp.ko 2023-07-20T18:29:05.816 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/sgi-xp/xpc.ko 2023-07-20T18:29:05.816 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-07-20T18:29:05.816 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/tifm_7xx1.ko 2023-07-20T18:29:05.817 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/tifm_core.ko 2023-07-20T18:29:05.817 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/tsl2550.ko 2023-07-20T18:29:05.817 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/vmw_balloon.ko 2023-07-20T18:29:05.817 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/vmw_vmci 2023-07-20T18:29:05.817 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-07-20T18:29:05.817 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net 2023-07-20T18:29:05.818 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/bonding 2023-07-20T18:29:05.818 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/bonding/bonding.ko 2023-07-20T18:29:05.818 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/dsa 2023-07-20T18:29:05.818 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/dsa/b53 2023-07-20T18:29:05.818 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/dsa/b53/b53_common.ko 2023-07-20T18:29:05.818 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/dsa/bcm-sf2.ko 2023-07-20T18:29:05.818 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/dsa/mv88e6060.ko 2023-07-20T18:29:05.819 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/dsa/mv88e6xxx 2023-07-20T18:29:05.819 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-07-20T18:29:05.819 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/dummy.ko 2023-07-20T18:29:05.819 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/eql.ko 2023-07-20T18:29:05.819 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet 2023-07-20T18:29:05.819 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/3com 2023-07-20T18:29:05.819 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-07-20T18:29:05.819 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-07-20T18:29:05.820 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-07-20T18:29:05.820 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-07-20T18:29:05.820 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/8390 2023-07-20T18:29:05.820 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/8390/8390.ko 2023-07-20T18:29:05.820 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-07-20T18:29:05.820 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-07-20T18:29:05.820 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-07-20T18:29:05.820 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/adaptec 2023-07-20T18:29:05.821 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-07-20T18:29:05.821 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/agere 2023-07-20T18:29:05.821 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/agere/et131x.ko 2023-07-20T18:29:05.821 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/alteon 2023-07-20T18:29:05.821 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-07-20T18:29:05.821 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/altera 2023-07-20T18:29:05.821 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-07-20T18:29:05.822 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/amd 2023-07-20T18:29:05.822 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-07-20T18:29:05.822 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-07-20T18:29:05.822 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-07-20T18:29:05.822 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/atheros 2023-07-20T18:29:05.822 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/atheros/alx 2023-07-20T18:29:05.822 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-07-20T18:29:05.822 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/atheros/atl1c 2023-07-20T18:29:05.823 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-07-20T18:29:05.823 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/atheros/atl1e 2023-07-20T18:29:05.823 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-07-20T18:29:05.823 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/atheros/atlx 2023-07-20T18:29:05.823 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-07-20T18:29:05.823 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-07-20T18:29:05.823 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/broadcom 2023-07-20T18:29:05.823 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-07-20T18:29:05.824 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-07-20T18:29:05.824 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-07-20T18:29:05.824 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-07-20T18:29:05.824 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-07-20T18:29:05.824 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/broadcom/genet 2023-07-20T18:29:05.824 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-07-20T18:29:05.824 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-07-20T18:29:05.825 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/brocade 2023-07-20T18:29:05.825 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/brocade/bna 2023-07-20T18:29:05.825 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-07-20T18:29:05.825 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio 2023-07-20T18:29:05.825 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio/cxgb 2023-07-20T18:29:05.825 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-07-20T18:29:05.825 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-07-20T18:29:05.825 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-07-20T18:29:05.826 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-07-20T18:29:05.826 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-07-20T18:29:05.826 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-07-20T18:29:05.826 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-07-20T18:29:05.826 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-07-20T18:29:05.826 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-07-20T18:29:05.826 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/cisco 2023-07-20T18:29:05.826 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/cisco/enic 2023-07-20T18:29:05.827 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-07-20T18:29:05.827 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dec 2023-07-20T18:29:05.827 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dec/tulip 2023-07-20T18:29:05.827 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-07-20T18:29:05.827 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-07-20T18:29:05.827 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-07-20T18:29:05.827 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-07-20T18:29:05.827 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-07-20T18:29:05.828 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-07-20T18:29:05.828 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dlink 2023-07-20T18:29:05.828 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-07-20T18:29:05.828 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-07-20T18:29:05.828 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/dnet.ko 2023-07-20T18:29:05.828 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/emulex 2023-07-20T18:29:05.828 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/emulex/benet 2023-07-20T18:29:05.829 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-07-20T18:29:05.829 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/ethoc.ko 2023-07-20T18:29:05.829 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/fealnx.ko 2023-07-20T18:29:05.829 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel 2023-07-20T18:29:05.829 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/e100.ko 2023-07-20T18:29:05.829 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/e1000 2023-07-20T18:29:05.829 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-07-20T18:29:05.829 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/e1000e 2023-07-20T18:29:05.830 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-07-20T18:29:05.830 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/fm10k 2023-07-20T18:29:05.830 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-07-20T18:29:05.830 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/i40e 2023-07-20T18:29:05.830 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-07-20T18:29:05.830 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/iavf 2023-07-20T18:29:05.830 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-07-20T18:29:05.830 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/igb 2023-07-20T18:29:05.831 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-07-20T18:29:05.831 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/igbvf 2023-07-20T18:29:05.831 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-07-20T18:29:05.831 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/ixgbe 2023-07-20T18:29:05.831 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-07-20T18:29:05.831 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/ixgbevf 2023-07-20T18:29:05.831 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-07-20T18:29:05.831 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/jme.ko 2023-07-20T18:29:05.832 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/marvell 2023-07-20T18:29:05.832 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-07-20T18:29:05.832 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/marvell/skge.ko 2023-07-20T18:29:05.832 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-07-20T18:29:05.832 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/mellanox 2023-07-20T18:29:05.832 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/mellanox/mlx4 2023-07-20T18:29:05.832 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-07-20T18:29:05.832 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-07-20T18:29:05.833 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/mellanox/mlx5 2023-07-20T18:29:05.833 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-07-20T18:29:05.833 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-07-20T18:29:05.833 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/micrel 2023-07-20T18:29:05.833 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-07-20T18:29:05.833 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/myricom 2023-07-20T18:29:05.833 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/myricom/myri10ge 2023-07-20T18:29:05.834 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-07-20T18:29:05.834 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/natsemi 2023-07-20T18:29:05.834 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-07-20T18:29:05.834 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-07-20T18:29:05.834 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/neterion 2023-07-20T18:29:05.834 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-07-20T18:29:05.834 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/nvidia 2023-07-20T18:29:05.834 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-07-20T18:29:05.835 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/packetengines 2023-07-20T18:29:05.835 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-07-20T18:29:05.835 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-07-20T18:29:05.835 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/qlogic 2023-07-20T18:29:05.835 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/qlogic/netxen 2023-07-20T18:29:05.835 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-07-20T18:29:05.835 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-07-20T18:29:05.835 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-07-20T18:29:05.836 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-07-20T18:29:05.836 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/rdc 2023-07-20T18:29:05.836 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-07-20T18:29:05.836 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/realtek 2023-07-20T18:29:05.836 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-07-20T18:29:05.836 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-07-20T18:29:05.836 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/realtek/atp.ko 2023-07-20T18:29:05.837 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-07-20T18:29:05.837 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/rocker 2023-07-20T18:29:05.837 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-07-20T18:29:05.837 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/sfc 2023-07-20T18:29:05.837 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-07-20T18:29:05.837 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/silan 2023-07-20T18:29:05.837 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-07-20T18:29:05.837 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/sis 2023-07-20T18:29:05.838 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/sis/sis190.ko 2023-07-20T18:29:05.838 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/sis/sis900.ko 2023-07-20T18:29:05.838 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/smsc 2023-07-20T18:29:05.838 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-07-20T18:29:05.838 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-07-20T18:29:05.838 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-07-20T18:29:05.838 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-07-20T18:29:05.839 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/stmicro 2023-07-20T18:29:05.839 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/stmicro/stmmac 2023-07-20T18:29:05.839 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-07-20T18:29:05.839 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-07-20T18:29:05.839 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/sun 2023-07-20T18:29:05.839 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/sun/cassini.ko 2023-07-20T18:29:05.839 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/sun/niu.ko 2023-07-20T18:29:05.839 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/sun/sungem.ko 2023-07-20T18:29:05.840 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-07-20T18:29:05.840 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/tehuti 2023-07-20T18:29:05.840 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-07-20T18:29:05.840 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/ti 2023-07-20T18:29:05.840 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/ti/tlan.ko 2023-07-20T18:29:05.840 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/via 2023-07-20T18:29:05.840 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-07-20T18:29:05.840 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-07-20T18:29:05.841 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/wiznet 2023-07-20T18:29:05.841 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-07-20T18:29:05.841 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-07-20T18:29:05.841 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/xircom 2023-07-20T18:29:05.841 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-07-20T18:29:05.841 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/geneve.ko 2023-07-20T18:29:05.841 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/hyperv 2023-07-20T18:29:05.842 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-07-20T18:29:05.842 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ieee802154 2023-07-20T18:29:05.842 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ieee802154/fakelb.ko 2023-07-20T18:29:05.842 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ifb.ko 2023-07-20T18:29:05.842 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ipvlan 2023-07-20T18:29:05.842 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ipvlan/ipvlan.ko 2023-07-20T18:29:05.842 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/macvlan.ko 2023-07-20T18:29:05.842 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/macvtap.ko 2023-07-20T18:29:05.843 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/mdio 2023-07-20T18:29:05.843 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/mdio.ko 2023-07-20T18:29:05.843 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-07-20T18:29:05.843 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-07-20T18:29:05.843 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/mdio/mdio-regmap.ko 2023-07-20T18:29:05.843 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/mii.ko 2023-07-20T18:29:05.843 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/net_failover.ko 2023-07-20T18:29:05.843 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/netconsole.ko 2023-07-20T18:29:05.844 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/nlmon.ko 2023-07-20T18:29:05.844 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/pcs 2023-07-20T18:29:05.844 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/pcs/pcs-lynx.ko 2023-07-20T18:29:05.844 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-07-20T18:29:05.844 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy 2023-07-20T18:29:05.844 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/amd.ko 2023-07-20T18:29:05.844 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-07-20T18:29:05.845 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-07-20T18:29:05.845 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/bcm7xxx.ko 2023-07-20T18:29:05.845 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/bcm87xx.ko 2023-07-20T18:29:05.845 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/broadcom.ko 2023-07-20T18:29:05.845 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/cicada.ko 2023-07-20T18:29:05.845 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/davicom.ko 2023-07-20T18:29:05.845 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/dp83640.ko 2023-07-20T18:29:05.845 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/et1011c.ko 2023-07-20T18:29:05.846 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/icplus.ko 2023-07-20T18:29:05.846 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/lxt.ko 2023-07-20T18:29:05.846 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/marvell.ko 2023-07-20T18:29:05.846 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/micrel.ko 2023-07-20T18:29:05.846 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/national.ko 2023-07-20T18:29:05.846 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/phylink.ko 2023-07-20T18:29:05.846 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/qsemi.ko 2023-07-20T18:29:05.846 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/realtek.ko 2023-07-20T18:29:05.847 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/smsc.ko 2023-07-20T18:29:05.847 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/ste10Xp.ko 2023-07-20T18:29:05.847 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/phy/vitesse.ko 2023-07-20T18:29:05.847 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ppp 2023-07-20T18:29:05.847 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ppp/bsd_comp.ko 2023-07-20T18:29:05.847 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ppp/ppp_async.ko 2023-07-20T18:29:05.847 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ppp/ppp_deflate.ko 2023-07-20T18:29:05.847 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ppp/ppp_generic.ko 2023-07-20T18:29:05.848 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ppp/ppp_mppe.ko 2023-07-20T18:29:05.848 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ppp/ppp_synctty.ko 2023-07-20T18:29:05.848 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ppp/pppoe.ko 2023-07-20T18:29:05.848 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ppp/pppox.ko 2023-07-20T18:29:05.848 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/ppp/pptp.ko 2023-07-20T18:29:05.848 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/slip 2023-07-20T18:29:05.848 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/slip/slhc.ko 2023-07-20T18:29:05.849 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/slip/slip.ko 2023-07-20T18:29:05.849 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/sungem_phy.ko 2023-07-20T18:29:05.849 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/tap.ko 2023-07-20T18:29:05.849 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/team 2023-07-20T18:29:05.849 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/team/team.ko 2023-07-20T18:29:05.849 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/team/team_mode_activebackup.ko 2023-07-20T18:29:05.849 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/team/team_mode_broadcast.ko 2023-07-20T18:29:05.849 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-07-20T18:29:05.850 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/team/team_mode_random.ko 2023-07-20T18:29:05.850 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-07-20T18:29:05.850 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/tun.ko 2023-07-20T18:29:05.850 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/veth.ko 2023-07-20T18:29:05.850 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/virtio_net.ko 2023-07-20T18:29:05.850 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/vmxnet3 2023-07-20T18:29:05.850 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-07-20T18:29:05.850 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/vxlan 2023-07-20T18:29:05.851 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/vxlan/vxlan.ko 2023-07-20T18:29:05.851 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/xen-netback 2023-07-20T18:29:05.851 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/xen-netback/xen-netback.ko 2023-07-20T18:29:05.851 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/net/xen-netfront.ko 2023-07-20T18:29:05.851 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ntb 2023-07-20T18:29:05.851 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ntb/ntb.ko 2023-07-20T18:29:05.851 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/nvme 2023-07-20T18:29:05.852 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/nvme/host 2023-07-20T18:29:05.852 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/nvme/host/nvme-core.ko 2023-07-20T18:29:05.852 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/nvme/host/nvme.ko 2023-07-20T18:29:05.852 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/parport 2023-07-20T18:29:05.852 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/parport/parport.ko 2023-07-20T18:29:05.852 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/parport/parport_cs.ko 2023-07-20T18:29:05.852 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/parport/parport_pc.ko 2023-07-20T18:29:05.852 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/parport/parport_serial.ko 2023-07-20T18:29:05.853 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pci 2023-07-20T18:29:05.853 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pci/hotplug 2023-07-20T18:29:05.853 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-07-20T18:29:05.853 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pci/pcie 2023-07-20T18:29:05.853 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pci/pcie/aer_inject.ko 2023-07-20T18:29:05.853 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pci/xen-pcifront.ko 2023-07-20T18:29:05.853 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pcmcia 2023-07-20T18:29:05.853 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pcmcia/i82092.ko 2023-07-20T18:29:05.854 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pcmcia/pd6729.ko 2023-07-20T18:29:05.854 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pcmcia/yenta_socket.ko 2023-07-20T18:29:05.854 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pinctrl 2023-07-20T18:29:05.854 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pinctrl/intel 2023-07-20T18:29:05.854 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-07-20T18:29:05.854 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform 2023-07-20T18:29:05.854 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86 2023-07-20T18:29:05.854 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/acer-wmi.ko 2023-07-20T18:29:05.855 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/acerhdf.ko 2023-07-20T18:29:05.855 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/apple-gmux.ko 2023-07-20T18:29:05.855 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/asus-laptop.ko 2023-07-20T18:29:05.855 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-07-20T18:29:05.855 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/asus-wmi.ko 2023-07-20T18:29:05.855 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/classmate-laptop.ko 2023-07-20T18:29:05.855 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-07-20T18:29:05.856 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-07-20T18:29:05.856 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-07-20T18:29:05.856 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-07-20T18:29:05.856 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/hdaps.ko 2023-07-20T18:29:05.856 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/intel 2023-07-20T18:29:05.856 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-07-20T18:29:05.856 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/intel_ips.ko 2023-07-20T18:29:05.856 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/msi-wmi.ko 2023-07-20T18:29:05.857 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/mxm-wmi.ko 2023-07-20T18:29:05.857 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-07-20T18:29:05.857 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/samsung-laptop.ko 2023-07-20T18:29:05.857 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/samsung-q10.ko 2023-07-20T18:29:05.857 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/topstar-laptop.ko 2023-07-20T18:29:05.857 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-07-20T18:29:05.857 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/toshiba_haps.ko 2023-07-20T18:29:05.857 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/wmi-bmof.ko 2023-07-20T18:29:05.858 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/platform/x86/wmi.ko 2023-07-20T18:29:05.858 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/powercap 2023-07-20T18:29:05.858 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/powercap/intel_rapl_common.ko 2023-07-20T18:29:05.858 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/powercap/intel_rapl_msr.ko 2023-07-20T18:29:05.858 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pps 2023-07-20T18:29:05.858 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pps/clients 2023-07-20T18:29:05.858 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pps/clients/pps-gpio.ko 2023-07-20T18:29:05.858 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pps/clients/pps-ldisc.ko 2023-07-20T18:29:05.859 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pps/clients/pps_parport.ko 2023-07-20T18:29:05.859 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/pps/pps_core.ko 2023-07-20T18:29:05.859 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ptp 2023-07-20T18:29:05.859 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ptp/ptp.ko 2023-07-20T18:29:05.859 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ptp/ptp_kvm.ko 2023-07-20T18:29:05.859 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc 2023-07-20T18:29:05.859 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-bq32k.ko 2023-07-20T18:29:05.860 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-bq4802.ko 2023-07-20T18:29:05.860 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-ds1286.ko 2023-07-20T18:29:05.860 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-ds1307.ko 2023-07-20T18:29:05.860 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-ds1374.ko 2023-07-20T18:29:05.860 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-ds1511.ko 2023-07-20T18:29:05.860 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-ds1553.ko 2023-07-20T18:29:05.860 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-ds1672.ko 2023-07-20T18:29:05.860 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-ds1742.ko 2023-07-20T18:29:05.861 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-ds2404.ko 2023-07-20T18:29:05.861 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-ds3232.ko 2023-07-20T18:29:05.861 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-em3027.ko 2023-07-20T18:29:05.861 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-fm3130.ko 2023-07-20T18:29:05.861 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-isl12022.ko 2023-07-20T18:29:05.861 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-isl1208.ko 2023-07-20T18:29:05.861 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-m41t80.ko 2023-07-20T18:29:05.861 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-m48t35.ko 2023-07-20T18:29:05.862 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-m48t59.ko 2023-07-20T18:29:05.862 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-max6900.ko 2023-07-20T18:29:05.862 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-msm6242.ko 2023-07-20T18:29:05.862 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-pcf2127.ko 2023-07-20T18:29:05.862 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-pcf85063.ko 2023-07-20T18:29:05.862 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-pcf8523.ko 2023-07-20T18:29:05.862 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-pcf8563.ko 2023-07-20T18:29:05.862 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-pcf8583.ko 2023-07-20T18:29:05.863 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-rp5c01.ko 2023-07-20T18:29:05.863 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-rs5c372.ko 2023-07-20T18:29:05.863 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-07-20T18:29:05.863 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-rx8025.ko 2023-07-20T18:29:05.863 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-rx8581.ko 2023-07-20T18:29:05.863 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-07-20T18:29:05.864 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/rtc/rtc-x1205.ko 2023-07-20T18:29:05.864 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi 2023-07-20T18:29:05.864 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/3w-9xxx.ko 2023-07-20T18:29:05.864 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/3w-sas.ko 2023-07-20T18:29:05.864 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/3w-xxxx.ko 2023-07-20T18:29:05.864 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/BusLogic.ko 2023-07-20T18:29:05.865 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/a100u2w.ko 2023-07-20T18:29:05.865 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/aacraid 2023-07-20T18:29:05.865 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/aacraid/aacraid.ko 2023-07-20T18:29:05.865 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/advansys.ko 2023-07-20T18:29:05.865 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/aic7xxx 2023-07-20T18:29:05.865 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-07-20T18:29:05.865 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-07-20T18:29:05.866 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/am53c974.ko 2023-07-20T18:29:05.866 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/arcmsr 2023-07-20T18:29:05.866 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-07-20T18:29:05.866 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/atp870u.ko 2023-07-20T18:29:05.866 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/be2iscsi 2023-07-20T18:29:05.866 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-07-20T18:29:05.866 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/bfa 2023-07-20T18:29:05.866 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/bfa/bfa.ko 2023-07-20T18:29:05.867 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/bnx2fc 2023-07-20T18:29:05.867 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-07-20T18:29:05.867 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/bnx2i 2023-07-20T18:29:05.867 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-07-20T18:29:05.867 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/ch.ko 2023-07-20T18:29:05.867 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/csiostor 2023-07-20T18:29:05.867 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/csiostor/csiostor.ko 2023-07-20T18:29:05.868 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/cxgbi 2023-07-20T18:29:05.868 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/cxgbi/cxgb3i 2023-07-20T18:29:05.868 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-07-20T18:29:05.868 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/cxgbi/cxgb4i 2023-07-20T18:29:05.868 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-07-20T18:29:05.868 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-07-20T18:29:05.868 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/dc395x.ko 2023-07-20T18:29:05.869 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/device_handler 2023-07-20T18:29:05.869 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-07-20T18:29:05.869 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-07-20T18:29:05.869 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-07-20T18:29:05.869 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-07-20T18:29:05.869 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/dmx3191d.ko 2023-07-20T18:29:05.869 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/esas2r 2023-07-20T18:29:05.869 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/esas2r/esas2r.ko 2023-07-20T18:29:05.870 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/esp_scsi.ko 2023-07-20T18:29:05.870 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/fcoe 2023-07-20T18:29:05.870 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/fcoe/fcoe.ko 2023-07-20T18:29:05.870 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-07-20T18:29:05.870 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/fnic 2023-07-20T18:29:05.870 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/fnic/fnic.ko 2023-07-20T18:29:05.870 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/hpsa.ko 2023-07-20T18:29:05.870 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/hptiop.ko 2023-07-20T18:29:05.871 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/hv_storvsc.ko 2023-07-20T18:29:05.871 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/initio.ko 2023-07-20T18:29:05.871 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/ipr.ko 2023-07-20T18:29:05.871 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/ips.ko 2023-07-20T18:29:05.871 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/isci 2023-07-20T18:29:05.871 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/isci/isci.ko 2023-07-20T18:29:05.871 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-07-20T18:29:05.871 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/iscsi_tcp.ko 2023-07-20T18:29:05.872 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/libfc 2023-07-20T18:29:05.872 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/libfc/libfc.ko 2023-07-20T18:29:05.872 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/libiscsi.ko 2023-07-20T18:29:05.872 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/libiscsi_tcp.ko 2023-07-20T18:29:05.872 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/libsas 2023-07-20T18:29:05.872 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/libsas/libsas.ko 2023-07-20T18:29:05.872 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/lpfc 2023-07-20T18:29:05.873 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/lpfc/lpfc.ko 2023-07-20T18:29:05.873 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/megaraid 2023-07-20T18:29:05.873 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/megaraid.ko 2023-07-20T18:29:05.873 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-07-20T18:29:05.873 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-07-20T18:29:05.873 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-07-20T18:29:05.873 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/mpt3sas 2023-07-20T18:29:05.873 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-07-20T18:29:05.874 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/mvsas 2023-07-20T18:29:05.874 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/mvsas/mvsas.ko 2023-07-20T18:29:05.874 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/mvumi.ko 2023-07-20T18:29:05.874 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/pm8001 2023-07-20T18:29:05.874 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-07-20T18:29:05.874 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/pmcraid.ko 2023-07-20T18:29:05.874 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/qla1280.ko 2023-07-20T18:29:05.874 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/qla2xxx 2023-07-20T18:29:05.875 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-07-20T18:29:05.875 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-07-20T18:29:05.875 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/qla4xxx 2023-07-20T18:29:05.875 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-07-20T18:29:05.875 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/raid_class.ko 2023-07-20T18:29:05.875 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/scsi_debug.ko 2023-07-20T18:29:05.875 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/scsi_transport_fc.ko 2023-07-20T18:29:05.875 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-07-20T18:29:05.876 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/scsi_transport_sas.ko 2023-07-20T18:29:05.876 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/scsi_transport_spi.ko 2023-07-20T18:29:05.876 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/scsi_transport_srp.ko 2023-07-20T18:29:05.876 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/ses.ko 2023-07-20T18:29:05.876 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/st.ko 2023-07-20T18:29:05.876 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/stex.ko 2023-07-20T18:29:05.876 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/sym53c8xx_2 2023-07-20T18:29:05.877 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-07-20T18:29:05.877 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/virtio_scsi.ko 2023-07-20T18:29:05.877 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/vmw_pvscsi.ko 2023-07-20T18:29:05.877 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/scsi/wd719x.ko 2023-07-20T18:29:05.877 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ssb 2023-07-20T18:29:05.877 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ssb/ssb.ko 2023-07-20T18:29:05.877 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target 2023-07-20T18:29:05.877 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/iscsi 2023-07-20T18:29:05.878 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-07-20T18:29:05.878 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/loopback 2023-07-20T18:29:05.878 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/loopback/tcm_loop.ko 2023-07-20T18:29:05.878 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/target_core_file.ko 2023-07-20T18:29:05.878 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/target_core_iblock.ko 2023-07-20T18:29:05.878 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/target_core_mod.ko 2023-07-20T18:29:05.878 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/target_core_pscsi.ko 2023-07-20T18:29:05.878 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/target_core_user.ko 2023-07-20T18:29:05.879 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/tcm_fc 2023-07-20T18:29:05.879 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-07-20T18:29:05.879 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal 2023-07-20T18:29:05.879 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel 2023-07-20T18:29:05.879 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal 2023-07-20T18:29:05.879 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-07-20T18:29:05.879 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-07-20T18:29:05.879 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-07-20T18:29:05.880 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-07-20T18:29:05.880 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-07-20T18:29:05.880 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-07-20T18:29:05.880 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-07-20T18:29:05.880 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-07-20T18:29:05.880 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-07-20T18:29:05.880 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-07-20T18:29:05.881 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-07-20T18:29:05.881 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-07-20T18:29:05.881 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-07-20T18:29:05.881 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-07-20T18:29:05.881 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-07-20T18:29:05.881 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty 2023-07-20T18:29:05.881 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/ipwireless 2023-07-20T18:29:05.881 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-07-20T18:29:05.882 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/n_gsm.ko 2023-07-20T18:29:05.882 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/n_hdlc.ko 2023-07-20T18:29:05.882 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/nozomi.ko 2023-07-20T18:29:05.882 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/serial 2023-07-20T18:29:05.882 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/serial/8250 2023-07-20T18:29:05.882 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-07-20T18:29:05.882 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/serial/arc_uart.ko 2023-07-20T18:29:05.883 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/serial/jsm 2023-07-20T18:29:05.883 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/serial/jsm/jsm.ko 2023-07-20T18:29:05.883 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/tty/synclink_gt.ko 2023-07-20T18:29:05.883 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ufs 2023-07-20T18:29:05.883 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ufs/core 2023-07-20T18:29:05.883 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ufs/core/ufshcd-core.ko 2023-07-20T18:29:05.883 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ufs/host 2023-07-20T18:29:05.883 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-07-20T18:29:05.884 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/uio 2023-07-20T18:29:05.884 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/uio/uio.ko 2023-07-20T18:29:05.884 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/uio/uio_aec.ko 2023-07-20T18:29:05.884 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/uio/uio_cif.ko 2023-07-20T18:29:05.884 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/uio/uio_pci_generic.ko 2023-07-20T18:29:05.884 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/uio/uio_sercos3.ko 2023-07-20T18:29:05.884 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb 2023-07-20T18:29:05.884 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/class 2023-07-20T18:29:05.885 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/class/cdc-acm.ko 2023-07-20T18:29:05.885 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/class/cdc-wdm.ko 2023-07-20T18:29:05.885 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/class/usblp.ko 2023-07-20T18:29:05.885 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/class/usbtmc.ko 2023-07-20T18:29:05.885 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/host 2023-07-20T18:29:05.885 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/host/sl811-hcd.ko 2023-07-20T18:29:05.885 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/image 2023-07-20T18:29:05.885 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/image/mdc800.ko 2023-07-20T18:29:05.886 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/image/microtek.ko 2023-07-20T18:29:05.886 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc 2023-07-20T18:29:05.886 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/adutux.ko 2023-07-20T18:29:05.886 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/appledisplay.ko 2023-07-20T18:29:05.886 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/emi26.ko 2023-07-20T18:29:05.886 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/emi62.ko 2023-07-20T18:29:05.886 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/ezusb.ko 2023-07-20T18:29:05.886 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/idmouse.ko 2023-07-20T18:29:05.887 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/iowarrior.ko 2023-07-20T18:29:05.887 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/isight_firmware.ko 2023-07-20T18:29:05.887 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/ldusb.ko 2023-07-20T18:29:05.887 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/legousbtower.ko 2023-07-20T18:29:05.887 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/sisusbvga 2023-07-20T18:29:05.887 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-07-20T18:29:05.887 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/trancevibrator.ko 2023-07-20T18:29:05.887 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/usb3503.ko 2023-07-20T18:29:05.888 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/usblcd.ko 2023-07-20T18:29:05.888 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/usbsevseg.ko 2023-07-20T18:29:05.888 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/uss720.ko 2023-07-20T18:29:05.888 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/misc/yurex.ko 2023-07-20T18:29:05.888 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/phy 2023-07-20T18:29:05.888 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/phy/phy-generic.ko 2023-07-20T18:29:05.888 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial 2023-07-20T18:29:05.889 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/aircable.ko 2023-07-20T18:29:05.889 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/ark3116.ko 2023-07-20T18:29:05.889 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/belkin_sa.ko 2023-07-20T18:29:05.889 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/ch341.ko 2023-07-20T18:29:05.889 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/cp210x.ko 2023-07-20T18:29:05.889 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/cyberjack.ko 2023-07-20T18:29:05.889 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/cypress_m8.ko 2023-07-20T18:29:05.889 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/digi_acceleport.ko 2023-07-20T18:29:05.890 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/empeg.ko 2023-07-20T18:29:05.890 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/ftdi_sio.ko 2023-07-20T18:29:05.890 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/garmin_gps.ko 2023-07-20T18:29:05.890 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/io_edgeport.ko 2023-07-20T18:29:05.890 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/io_ti.ko 2023-07-20T18:29:05.890 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/ipaq.ko 2023-07-20T18:29:05.890 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/ipw.ko 2023-07-20T18:29:05.890 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/ir-usb.ko 2023-07-20T18:29:05.891 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-07-20T18:29:05.891 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/keyspan.ko 2023-07-20T18:29:05.891 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/keyspan_pda.ko 2023-07-20T18:29:05.891 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/kl5kusb105.ko 2023-07-20T18:29:05.891 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/kobil_sct.ko 2023-07-20T18:29:05.891 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/mct_u232.ko 2023-07-20T18:29:05.891 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/mos7720.ko 2023-07-20T18:29:05.891 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/mos7840.ko 2023-07-20T18:29:05.892 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/navman.ko 2023-07-20T18:29:05.892 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/omninet.ko 2023-07-20T18:29:05.892 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/opticon.ko 2023-07-20T18:29:05.892 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/option.ko 2023-07-20T18:29:05.892 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/oti6858.ko 2023-07-20T18:29:05.892 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/pl2303.ko 2023-07-20T18:29:05.892 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/qcaux.ko 2023-07-20T18:29:05.892 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/qcserial.ko 2023-07-20T18:29:05.893 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/quatech2.ko 2023-07-20T18:29:05.893 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/safe_serial.ko 2023-07-20T18:29:05.893 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/sierra.ko 2023-07-20T18:29:05.893 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/spcp8x5.ko 2023-07-20T18:29:05.893 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/ssu100.ko 2023-07-20T18:29:05.893 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/symbolserial.ko 2023-07-20T18:29:05.893 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-07-20T18:29:05.893 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-07-20T18:29:05.894 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/usb_debug.ko 2023-07-20T18:29:05.894 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/usb_wwan.ko 2023-07-20T18:29:05.894 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/visor.ko 2023-07-20T18:29:05.894 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/whiteheat.ko 2023-07-20T18:29:05.894 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/serial/xsens_mt.ko 2023-07-20T18:29:05.894 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage 2023-07-20T18:29:05.894 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/uas.ko 2023-07-20T18:29:05.895 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-alauda.ko 2023-07-20T18:29:05.895 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-cypress.ko 2023-07-20T18:29:05.895 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-datafab.ko 2023-07-20T18:29:05.895 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-07-20T18:29:05.895 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-freecom.ko 2023-07-20T18:29:05.895 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-isd200.ko 2023-07-20T18:29:05.895 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-07-20T18:29:05.896 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-karma.ko 2023-07-20T18:29:05.896 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-onetouch.ko 2023-07-20T18:29:05.896 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-realtek.ko 2023-07-20T18:29:05.896 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-sddr09.ko 2023-07-20T18:29:05.896 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-sddr55.ko 2023-07-20T18:29:05.896 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/ums-usbat.ko 2023-07-20T18:29:05.896 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/storage/usb-storage.ko 2023-07-20T18:29:05.896 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/usbip 2023-07-20T18:29:05.897 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/usbip/usbip-core.ko 2023-07-20T18:29:05.897 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/usbip/usbip-host.ko 2023-07-20T18:29:05.897 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-07-20T18:29:05.897 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vfio 2023-07-20T18:29:05.897 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vfio/pci 2023-07-20T18:29:05.897 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-07-20T18:29:05.897 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vfio/pci/vfio-pci.ko 2023-07-20T18:29:05.898 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vfio/vfio.ko 2023-07-20T18:29:05.898 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-07-20T18:29:05.898 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vhost 2023-07-20T18:29:05.898 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vhost/vhost.ko 2023-07-20T18:29:05.898 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vhost/vhost_iotlb.ko 2023-07-20T18:29:05.899 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vhost/vhost_net.ko 2023-07-20T18:29:05.899 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/vhost/vhost_scsi.ko 2023-07-20T18:29:05.899 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/video 2023-07-20T18:29:05.899 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/video/backlight 2023-07-20T18:29:05.899 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/video/backlight/apple_bl.ko 2023-07-20T18:29:05.899 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/video/backlight/lcd.ko 2023-07-20T18:29:05.899 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/video/backlight/platform_lcd.ko 2023-07-20T18:29:05.900 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/virtio 2023-07-20T18:29:05.900 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/virtio/virtio.ko 2023-07-20T18:29:05.900 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/virtio/virtio_balloon.ko 2023-07-20T18:29:05.900 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/virtio/virtio_mmio.ko 2023-07-20T18:29:05.900 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/virtio/virtio_pci.ko 2023-07-20T18:29:05.900 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-07-20T18:29:05.900 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-07-20T18:29:05.900 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/virtio/virtio_ring.ko 2023-07-20T18:29:05.901 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1 2023-07-20T18:29:05.901 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/masters 2023-07-20T18:29:05.901 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/masters/ds2482.ko 2023-07-20T18:29:05.901 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/masters/ds2490.ko 2023-07-20T18:29:05.901 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves 2023-07-20T18:29:05.901 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-07-20T18:29:05.901 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-07-20T18:29:05.901 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-07-20T18:29:05.902 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-07-20T18:29:05.902 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-07-20T18:29:05.902 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-07-20T18:29:05.902 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-07-20T18:29:05.902 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-07-20T18:29:05.902 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-07-20T18:29:05.902 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_smem.ko 2023-07-20T18:29:05.902 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/slaves/w1_therm.ko 2023-07-20T18:29:05.903 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/w1/wire.ko 2023-07-20T18:29:05.903 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog 2023-07-20T18:29:05.903 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/alim1535_wdt.ko 2023-07-20T18:29:05.903 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/alim7101_wdt.ko 2023-07-20T18:29:05.903 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/f71808e_wdt.ko 2023-07-20T18:29:05.903 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/hpwdt.ko 2023-07-20T18:29:05.903 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/i6300esb.ko 2023-07-20T18:29:05.903 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-07-20T18:29:05.904 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/iTCO_wdt.ko 2023-07-20T18:29:05.904 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/ib700wdt.ko 2023-07-20T18:29:05.904 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/ibmasr.ko 2023-07-20T18:29:05.904 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-07-20T18:29:05.904 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/it8712f_wdt.ko 2023-07-20T18:29:05.904 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/it87_wdt.ko 2023-07-20T18:29:05.904 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/machzwd.ko 2023-07-20T18:29:05.905 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/nv_tco.ko 2023-07-20T18:29:05.905 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/pcwd_pci.ko 2023-07-20T18:29:05.905 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/pcwd_usb.ko 2023-07-20T18:29:05.905 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-07-20T18:29:05.905 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/sch311x_wdt.ko 2023-07-20T18:29:05.905 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/softdog.ko 2023-07-20T18:29:05.905 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/sp5100_tco.ko 2023-07-20T18:29:05.905 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/via_wdt.ko 2023-07-20T18:29:05.906 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-07-20T18:29:05.906 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/w83877f_wdt.ko 2023-07-20T18:29:05.906 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/w83977f_wdt.ko 2023-07-20T18:29:05.906 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/wdt_pci.ko 2023-07-20T18:29:05.906 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/watchdog/xen_wdt.ko 2023-07-20T18:29:05.906 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/xen 2023-07-20T18:29:05.906 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/xen/xen-acpi-processor.ko 2023-07-20T18:29:05.906 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/xen/xen-evtchn.ko 2023-07-20T18:29:05.907 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/xen/xen-gntalloc.ko 2023-07-20T18:29:05.907 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/xen/xen-gntdev.ko 2023-07-20T18:29:05.907 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/xen/xen-pciback 2023-07-20T18:29:05.907 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-07-20T18:29:05.907 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/xen/xen-privcmd.ko 2023-07-20T18:29:05.907 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/xen/xenfs 2023-07-20T18:29:05.907 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/drivers/xen/xenfs/xenfs.ko 2023-07-20T18:29:05.907 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs 2023-07-20T18:29:05.908 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/9p 2023-07-20T18:29:05.908 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/9p/9p.ko 2023-07-20T18:29:05.908 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/affs 2023-07-20T18:29:05.908 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/affs/affs.ko 2023-07-20T18:29:05.908 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/befs 2023-07-20T18:29:05.908 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/befs/befs.ko 2023-07-20T18:29:05.908 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/binfmt_misc.ko 2023-07-20T18:29:05.908 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/btrfs 2023-07-20T18:29:05.909 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/btrfs/btrfs.ko 2023-07-20T18:29:05.909 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/cachefiles 2023-07-20T18:29:05.909 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/cachefiles/cachefiles.ko 2023-07-20T18:29:05.909 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ceph 2023-07-20T18:29:05.909 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ceph/ceph.ko 2023-07-20T18:29:05.909 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/coda 2023-07-20T18:29:05.909 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/coda/coda.ko 2023-07-20T18:29:05.909 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/cramfs 2023-07-20T18:29:05.910 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/cramfs/cramfs.ko 2023-07-20T18:29:05.910 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/dlm 2023-07-20T18:29:05.910 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/dlm/dlm.ko 2023-07-20T18:29:05.910 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ecryptfs 2023-07-20T18:29:05.910 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ecryptfs/ecryptfs.ko 2023-07-20T18:29:05.910 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/f2fs 2023-07-20T18:29:05.910 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/f2fs/f2fs.ko 2023-07-20T18:29:05.910 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/fat 2023-07-20T18:29:05.911 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/fat/fat.ko 2023-07-20T18:29:05.911 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/fat/msdos.ko 2023-07-20T18:29:05.911 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/fat/vfat.ko 2023-07-20T18:29:05.911 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/fscache 2023-07-20T18:29:05.911 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/fscache/fscache.ko 2023-07-20T18:29:05.911 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/fuse 2023-07-20T18:29:05.911 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/fuse/cuse.ko 2023-07-20T18:29:05.912 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/fuse/fuse.ko 2023-07-20T18:29:05.912 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/gfs2 2023-07-20T18:29:05.912 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/gfs2/gfs2.ko 2023-07-20T18:29:05.912 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/hfs 2023-07-20T18:29:05.912 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/hfs/hfs.ko 2023-07-20T18:29:05.912 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/hfsplus 2023-07-20T18:29:05.912 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/hfsplus/hfsplus.ko 2023-07-20T18:29:05.912 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/isofs 2023-07-20T18:29:05.913 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/isofs/isofs.ko 2023-07-20T18:29:05.913 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/jfs 2023-07-20T18:29:05.913 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/jfs/jfs.ko 2023-07-20T18:29:05.913 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/lockd 2023-07-20T18:29:05.913 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/lockd/lockd.ko 2023-07-20T18:29:05.913 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/minix 2023-07-20T18:29:05.913 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/minix/minix.ko 2023-07-20T18:29:05.914 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/netfs 2023-07-20T18:29:05.914 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/netfs/netfs.ko 2023-07-20T18:29:05.914 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs 2023-07-20T18:29:05.914 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs/blocklayout 2023-07-20T18:29:05.914 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-07-20T18:29:05.914 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs/filelayout 2023-07-20T18:29:05.914 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-07-20T18:29:05.914 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs/flexfilelayout 2023-07-20T18:29:05.915 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-07-20T18:29:05.915 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs/nfs.ko 2023-07-20T18:29:05.915 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs/nfsv3.ko 2023-07-20T18:29:05.915 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs/nfsv4.ko 2023-07-20T18:29:05.915 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs_common 2023-07-20T18:29:05.915 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs_common/grace.ko 2023-07-20T18:29:05.915 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfs_common/nfs_acl.ko 2023-07-20T18:29:05.915 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfsd 2023-07-20T18:29:05.916 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nfsd/nfsd.ko 2023-07-20T18:29:05.916 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nilfs2 2023-07-20T18:29:05.916 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nilfs2/nilfs2.ko 2023-07-20T18:29:05.916 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls 2023-07-20T18:29:05.916 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-celtic.ko 2023-07-20T18:29:05.916 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-centeuro.ko 2023-07-20T18:29:05.916 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-croatian.ko 2023-07-20T18:29:05.916 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-cyrillic.ko 2023-07-20T18:29:05.917 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-gaelic.ko 2023-07-20T18:29:05.917 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-greek.ko 2023-07-20T18:29:05.917 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-iceland.ko 2023-07-20T18:29:05.917 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-inuit.ko 2023-07-20T18:29:05.917 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-roman.ko 2023-07-20T18:29:05.917 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-romanian.ko 2023-07-20T18:29:05.917 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/mac-turkish.ko 2023-07-20T18:29:05.918 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp1250.ko 2023-07-20T18:29:05.918 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp1251.ko 2023-07-20T18:29:05.918 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp1255.ko 2023-07-20T18:29:05.918 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp737.ko 2023-07-20T18:29:05.918 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp775.ko 2023-07-20T18:29:05.918 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp850.ko 2023-07-20T18:29:05.918 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp852.ko 2023-07-20T18:29:05.918 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp855.ko 2023-07-20T18:29:05.919 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp857.ko 2023-07-20T18:29:05.919 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp860.ko 2023-07-20T18:29:05.919 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp861.ko 2023-07-20T18:29:05.919 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp862.ko 2023-07-20T18:29:05.919 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp863.ko 2023-07-20T18:29:05.919 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp864.ko 2023-07-20T18:29:05.919 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp865.ko 2023-07-20T18:29:05.919 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp866.ko 2023-07-20T18:29:05.920 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp869.ko 2023-07-20T18:29:05.920 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp874.ko 2023-07-20T18:29:05.920 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp932.ko 2023-07-20T18:29:05.920 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp936.ko 2023-07-20T18:29:05.920 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp949.ko 2023-07-20T18:29:05.920 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_cp950.ko 2023-07-20T18:29:05.920 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_euc-jp.ko 2023-07-20T18:29:05.920 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-1.ko 2023-07-20T18:29:05.921 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-13.ko 2023-07-20T18:29:05.921 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-14.ko 2023-07-20T18:29:05.921 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-15.ko 2023-07-20T18:29:05.921 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-2.ko 2023-07-20T18:29:05.921 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-3.ko 2023-07-20T18:29:05.921 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-4.ko 2023-07-20T18:29:05.921 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-5.ko 2023-07-20T18:29:05.921 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-6.ko 2023-07-20T18:29:05.922 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-7.ko 2023-07-20T18:29:05.922 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_iso8859-9.ko 2023-07-20T18:29:05.922 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_koi8-r.ko 2023-07-20T18:29:05.922 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_koi8-ru.ko 2023-07-20T18:29:05.922 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_koi8-u.ko 2023-07-20T18:29:05.922 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/nls/nls_utf8.ko 2023-07-20T18:29:05.922 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2 2023-07-20T18:29:05.922 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2/cluster 2023-07-20T18:29:05.923 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-07-20T18:29:05.923 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2/dlm 2023-07-20T18:29:05.923 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-07-20T18:29:05.923 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2/dlmfs 2023-07-20T18:29:05.923 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-07-20T18:29:05.923 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2/ocfs2.ko 2023-07-20T18:29:05.923 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-07-20T18:29:05.923 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-07-20T18:29:05.924 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-07-20T18:29:05.924 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/overlayfs 2023-07-20T18:29:05.924 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/overlayfs/overlay.ko 2023-07-20T18:29:05.924 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/pstore 2023-07-20T18:29:05.924 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/pstore/ramoops.ko 2023-07-20T18:29:05.924 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/reiserfs 2023-07-20T18:29:05.924 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/reiserfs/reiserfs.ko 2023-07-20T18:29:05.925 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/romfs 2023-07-20T18:29:05.925 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/romfs/romfs.ko 2023-07-20T18:29:05.925 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/smb 2023-07-20T18:29:05.925 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/smb/client 2023-07-20T18:29:05.925 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/smb/client/cifs.ko 2023-07-20T18:29:05.925 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/smb/common 2023-07-20T18:29:05.925 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/smb/common/cifs_arc4.ko 2023-07-20T18:29:05.925 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/smb/common/cifs_md4.ko 2023-07-20T18:29:05.926 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/squashfs 2023-07-20T18:29:05.926 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/squashfs/squashfs.ko 2023-07-20T18:29:05.926 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/sysv 2023-07-20T18:29:05.926 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/sysv/sysv.ko 2023-07-20T18:29:05.926 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/udf 2023-07-20T18:29:05.926 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/udf/udf.ko 2023-07-20T18:29:05.926 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ufs 2023-07-20T18:29:05.926 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/ufs/ufs.ko 2023-07-20T18:29:05.927 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/xfs 2023-07-20T18:29:05.927 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/fs/xfs/xfs.ko 2023-07-20T18:29:05.927 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/kernel 2023-07-20T18:29:05.927 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/kernel/rcu 2023-07-20T18:29:05.927 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/kernel/rcu/rcutorture.ko 2023-07-20T18:29:05.927 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/kernel/torture.ko 2023-07-20T18:29:05.927 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/kernel/trace 2023-07-20T18:29:05.927 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-07-20T18:29:05.928 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib 2023-07-20T18:29:05.928 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/asn1_encoder.ko 2023-07-20T18:29:05.928 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/crc-itu-t.ko 2023-07-20T18:29:05.928 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/crc8.ko 2023-07-20T18:29:05.928 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/crypto 2023-07-20T18:29:05.928 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/crypto/libarc4.ko 2023-07-20T18:29:05.928 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/crypto/libdes.ko 2023-07-20T18:29:05.929 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/libcrc32c.ko 2023-07-20T18:29:05.929 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/lru_cache.ko 2023-07-20T18:29:05.929 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/lz4 2023-07-20T18:29:05.929 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/lz4/lz4_compress.ko 2023-07-20T18:29:05.929 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/lz4/lz4hc_compress.ko 2023-07-20T18:29:05.929 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/math 2023-07-20T18:29:05.929 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/math/cordic.ko 2023-07-20T18:29:05.929 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/raid6 2023-07-20T18:29:05.930 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/raid6/raid6_pq.ko 2023-07-20T18:29:05.930 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/reed_solomon 2023-07-20T18:29:05.930 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/reed_solomon/reed_solomon.ko 2023-07-20T18:29:05.930 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/ts_bm.ko 2023-07-20T18:29:05.930 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/ts_fsm.ko 2023-07-20T18:29:05.930 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/ts_kmp.ko 2023-07-20T18:29:05.930 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/zstd 2023-07-20T18:29:05.930 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/lib/zstd/zstd_compress.ko 2023-07-20T18:29:05.931 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/mm 2023-07-20T18:29:05.931 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/mm/hwpoison-inject.ko 2023-07-20T18:29:05.931 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net 2023-07-20T18:29:05.931 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/6lowpan 2023-07-20T18:29:05.931 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/6lowpan/6lowpan.ko 2023-07-20T18:29:05.931 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/6lowpan/nhc_dest.ko 2023-07-20T18:29:05.931 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/6lowpan/nhc_fragment.ko 2023-07-20T18:29:05.931 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/6lowpan/nhc_hop.ko 2023-07-20T18:29:05.932 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/6lowpan/nhc_ipv6.ko 2023-07-20T18:29:05.932 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/6lowpan/nhc_mobility.ko 2023-07-20T18:29:05.932 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/6lowpan/nhc_routing.ko 2023-07-20T18:29:05.932 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/6lowpan/nhc_udp.ko 2023-07-20T18:29:05.932 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/802 2023-07-20T18:29:05.932 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/802/garp.ko 2023-07-20T18:29:05.932 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/802/mrp.ko 2023-07-20T18:29:05.932 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/802/p8022.ko 2023-07-20T18:29:05.933 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/802/psnap.ko 2023-07-20T18:29:05.933 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/802/stp.ko 2023-07-20T18:29:05.933 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/8021q 2023-07-20T18:29:05.933 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/8021q/8021q.ko 2023-07-20T18:29:05.933 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/9p 2023-07-20T18:29:05.933 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/9p/9pnet.ko 2023-07-20T18:29:05.933 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/9p/9pnet_fd.ko 2023-07-20T18:29:05.933 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/9p/9pnet_rdma.ko 2023-07-20T18:29:05.934 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/9p/9pnet_virtio.ko 2023-07-20T18:29:05.934 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/batman-adv 2023-07-20T18:29:05.934 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/batman-adv/batman-adv.ko 2023-07-20T18:29:05.934 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge 2023-07-20T18:29:05.934 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/br_netfilter.ko 2023-07-20T18:29:05.934 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/bridge.ko 2023-07-20T18:29:05.934 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter 2023-07-20T18:29:05.934 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-07-20T18:29:05.935 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_among.ko 2023-07-20T18:29:05.935 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_arp.ko 2023-07-20T18:29:05.935 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-07-20T18:29:05.935 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-07-20T18:29:05.935 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_ip.ko 2023-07-20T18:29:05.935 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-07-20T18:29:05.935 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_limit.ko 2023-07-20T18:29:05.935 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_log.ko 2023-07-20T18:29:05.936 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_mark.ko 2023-07-20T18:29:05.936 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-07-20T18:29:05.936 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-07-20T18:29:05.936 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-07-20T18:29:05.936 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-07-20T18:29:05.936 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_snat.ko 2023-07-20T18:29:05.936 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_stp.ko 2023-07-20T18:29:05.937 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-07-20T18:29:05.937 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-07-20T18:29:05.937 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-07-20T18:29:05.937 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-07-20T18:29:05.937 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/ebtables.ko 2023-07-20T18:29:05.937 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-07-20T18:29:05.937 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-07-20T18:29:05.937 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-07-20T18:29:05.938 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ceph 2023-07-20T18:29:05.938 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ceph/libceph.ko 2023-07-20T18:29:05.938 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/core 2023-07-20T18:29:05.938 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/core/failover.ko 2023-07-20T18:29:05.938 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/core/pktgen.ko 2023-07-20T18:29:05.938 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dccp 2023-07-20T18:29:05.938 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dccp/dccp.ko 2023-07-20T18:29:05.938 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dccp/dccp_diag.ko 2023-07-20T18:29:05.939 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dccp/dccp_ipv4.ko 2023-07-20T18:29:05.939 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dccp/dccp_ipv6.ko 2023-07-20T18:29:05.939 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dns_resolver 2023-07-20T18:29:05.939 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dns_resolver/dns_resolver.ko 2023-07-20T18:29:05.939 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dsa 2023-07-20T18:29:05.939 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dsa/dsa_core.ko 2023-07-20T18:29:05.939 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dsa/tag_brcm.ko 2023-07-20T18:29:05.939 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dsa/tag_dsa.ko 2023-07-20T18:29:05.940 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dsa/tag_none.ko 2023-07-20T18:29:05.940 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/dsa/tag_trailer.ko 2023-07-20T18:29:05.940 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ieee802154 2023-07-20T18:29:05.940 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ieee802154/6lowpan 2023-07-20T18:29:05.940 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-07-20T18:29:05.940 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ieee802154/ieee802154.ko 2023-07-20T18:29:05.940 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ieee802154/ieee802154_socket.ko 2023-07-20T18:29:05.940 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4 2023-07-20T18:29:05.941 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/ah4.ko 2023-07-20T18:29:05.941 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/esp4.ko 2023-07-20T18:29:05.941 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/fou.ko 2023-07-20T18:29:05.941 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/gre.ko 2023-07-20T18:29:05.941 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/inet_diag.ko 2023-07-20T18:29:05.941 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/ip_gre.ko 2023-07-20T18:29:05.941 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/ip_tunnel.ko 2023-07-20T18:29:05.942 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/ip_vti.ko 2023-07-20T18:29:05.942 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/ipcomp.ko 2023-07-20T18:29:05.942 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/ipip.ko 2023-07-20T18:29:05.942 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter 2023-07-20T18:29:05.942 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/arp_tables.ko 2023-07-20T18:29:05.942 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-07-20T18:29:05.942 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-07-20T18:29:05.942 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-07-20T18:29:05.943 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-07-20T18:29:05.943 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-07-20T18:29:05.943 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-07-20T18:29:05.943 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-07-20T18:29:05.943 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-07-20T18:29:05.943 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-07-20T18:29:05.943 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/iptable_security.ko 2023-07-20T18:29:05.943 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-07-20T18:29:05.944 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-07-20T18:29:05.944 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-07-20T18:29:05.944 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-07-20T18:29:05.944 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-07-20T18:29:05.944 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-07-20T18:29:05.944 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-07-20T18:29:05.944 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-07-20T18:29:05.944 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-07-20T18:29:05.945 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-07-20T18:29:05.945 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_bic.ko 2023-07-20T18:29:05.945 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_dctcp.ko 2023-07-20T18:29:05.945 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_diag.ko 2023-07-20T18:29:05.945 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_highspeed.ko 2023-07-20T18:29:05.945 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_htcp.ko 2023-07-20T18:29:05.945 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_hybla.ko 2023-07-20T18:29:05.946 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_illinois.ko 2023-07-20T18:29:05.946 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_lp.ko 2023-07-20T18:29:05.946 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_scalable.ko 2023-07-20T18:29:05.946 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_vegas.ko 2023-07-20T18:29:05.946 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_veno.ko 2023-07-20T18:29:05.946 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_westwood.ko 2023-07-20T18:29:05.946 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tcp_yeah.ko 2023-07-20T18:29:05.946 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/tunnel4.ko 2023-07-20T18:29:05.947 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/udp_diag.ko 2023-07-20T18:29:05.947 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/udp_tunnel.ko 2023-07-20T18:29:05.947 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv4/xfrm4_tunnel.ko 2023-07-20T18:29:05.947 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6 2023-07-20T18:29:05.947 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/ah6.ko 2023-07-20T18:29:05.947 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/esp6.ko 2023-07-20T18:29:05.947 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/fou6.ko 2023-07-20T18:29:05.947 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/ip6_tunnel.ko 2023-07-20T18:29:05.948 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-07-20T18:29:05.948 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/ip6_vti.ko 2023-07-20T18:29:05.948 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/ipcomp6.ko 2023-07-20T18:29:05.948 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter 2023-07-20T18:29:05.948 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-07-20T18:29:05.948 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-07-20T18:29:05.948 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-07-20T18:29:05.948 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-07-20T18:29:05.949 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-07-20T18:29:05.949 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-07-20T18:29:05.949 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-07-20T18:29:05.949 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-07-20T18:29:05.949 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-07-20T18:29:05.949 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-07-20T18:29:05.949 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-07-20T18:29:05.949 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-07-20T18:29:05.950 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-07-20T18:29:05.950 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-07-20T18:29:05.950 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-07-20T18:29:05.950 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-07-20T18:29:05.950 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-07-20T18:29:05.950 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-07-20T18:29:05.950 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-07-20T18:29:05.950 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-07-20T18:29:05.951 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-07-20T18:29:05.951 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-07-20T18:29:05.951 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-07-20T18:29:05.951 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-07-20T18:29:05.951 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/sit.ko 2023-07-20T18:29:05.951 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/tunnel6.ko 2023-07-20T18:29:05.951 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/ipv6/xfrm6_tunnel.ko 2023-07-20T18:29:05.951 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/key 2023-07-20T18:29:05.952 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/key/af_key.ko 2023-07-20T18:29:05.952 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/l2tp 2023-07-20T18:29:05.952 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/l2tp/l2tp_core.ko 2023-07-20T18:29:05.952 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/l2tp/l2tp_debugfs.ko 2023-07-20T18:29:05.952 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/l2tp/l2tp_eth.ko 2023-07-20T18:29:05.952 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/l2tp/l2tp_ip.ko 2023-07-20T18:29:05.952 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/l2tp/l2tp_ip6.ko 2023-07-20T18:29:05.953 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/l2tp/l2tp_netlink.ko 2023-07-20T18:29:05.953 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/l2tp/l2tp_ppp.ko 2023-07-20T18:29:05.953 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/llc 2023-07-20T18:29:05.953 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/llc/llc.ko 2023-07-20T18:29:05.953 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/mac802154 2023-07-20T18:29:05.953 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/mac802154/mac802154.ko 2023-07-20T18:29:05.953 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/mpls 2023-07-20T18:29:05.953 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/mpls/mpls_gso.ko 2023-07-20T18:29:05.954 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter 2023-07-20T18:29:05.954 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset 2023-07-20T18:29:05.954 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set.ko 2023-07-20T18:29:05.954 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-07-20T18:29:05.954 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-07-20T18:29:05.954 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-07-20T18:29:05.954 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-07-20T18:29:05.954 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-07-20T18:29:05.955 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-07-20T18:29:05.955 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-07-20T18:29:05.955 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-07-20T18:29:05.955 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-07-20T18:29:05.955 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-07-20T18:29:05.955 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-07-20T18:29:05.955 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-07-20T18:29:05.955 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-07-20T18:29:05.956 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-07-20T18:29:05.956 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-07-20T18:29:05.956 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs 2023-07-20T18:29:05.956 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs.ko 2023-07-20T18:29:05.956 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-07-20T18:29:05.956 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-07-20T18:29:05.956 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-07-20T18:29:05.956 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-07-20T18:29:05.957 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-07-20T18:29:05.957 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-07-20T18:29:05.957 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-07-20T18:29:05.957 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-07-20T18:29:05.957 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-07-20T18:29:05.957 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-07-20T18:29:05.957 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-07-20T18:29:05.957 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-07-20T18:29:05.958 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-07-20T18:29:05.958 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conncount.ko 2023-07-20T18:29:05.958 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack.ko 2023-07-20T18:29:05.958 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-07-20T18:29:05.958 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-07-20T18:29:05.958 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-07-20T18:29:05.958 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_h323.ko 2023-07-20T18:29:05.959 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_irc.ko 2023-07-20T18:29:05.959 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-07-20T18:29:05.959 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-07-20T18:29:05.959 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-07-20T18:29:05.959 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_sane.ko 2023-07-20T18:29:05.959 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_sip.ko 2023-07-20T18:29:05.959 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-07-20T18:29:05.959 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-07-20T18:29:05.960 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_dup_netdev.ko 2023-07-20T18:29:05.960 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_flow_table.ko 2023-07-20T18:29:05.960 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_flow_table_inet.ko 2023-07-20T18:29:05.960 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_log_syslog.ko 2023-07-20T18:29:05.960 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_nat.ko 2023-07-20T18:29:05.960 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_nat_amanda.ko 2023-07-20T18:29:05.960 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_nat_ftp.ko 2023-07-20T18:29:05.960 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_nat_irc.ko 2023-07-20T18:29:05.961 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_nat_sip.ko 2023-07-20T18:29:05.961 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_nat_tftp.ko 2023-07-20T18:29:05.961 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_synproxy_core.ko 2023-07-20T18:29:05.961 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nf_tables.ko 2023-07-20T18:29:05.961 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nfnetlink.ko 2023-07-20T18:29:05.961 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nfnetlink_acct.ko 2023-07-20T18:29:05.961 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nfnetlink_log.ko 2023-07-20T18:29:05.961 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nfnetlink_osf.ko 2023-07-20T18:29:05.962 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nfnetlink_queue.ko 2023-07-20T18:29:05.962 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_chain_nat.ko 2023-07-20T18:29:05.962 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_compat.ko 2023-07-20T18:29:05.962 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_ct.ko 2023-07-20T18:29:05.962 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_dup_netdev.ko 2023-07-20T18:29:05.962 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_fib.ko 2023-07-20T18:29:05.962 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_fwd_netdev.ko 2023-07-20T18:29:05.962 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_hash.ko 2023-07-20T18:29:05.963 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_limit.ko 2023-07-20T18:29:05.963 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_log.ko 2023-07-20T18:29:05.963 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_masq.ko 2023-07-20T18:29:05.963 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_nat.ko 2023-07-20T18:29:05.963 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_numgen.ko 2023-07-20T18:29:05.963 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_queue.ko 2023-07-20T18:29:05.963 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_quota.ko 2023-07-20T18:29:05.963 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_redir.ko 2023-07-20T18:29:05.964 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_reject.ko 2023-07-20T18:29:05.964 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_reject_inet.ko 2023-07-20T18:29:05.964 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_socket.ko 2023-07-20T18:29:05.964 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/nft_tunnel.ko 2023-07-20T18:29:05.964 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_AUDIT.ko 2023-07-20T18:29:05.964 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_CHECKSUM.ko 2023-07-20T18:29:05.964 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_CLASSIFY.ko 2023-07-20T18:29:05.965 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-07-20T18:29:05.965 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_CT.ko 2023-07-20T18:29:05.965 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_DSCP.ko 2023-07-20T18:29:05.965 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_HL.ko 2023-07-20T18:29:05.965 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_HMARK.ko 2023-07-20T18:29:05.965 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_IDLETIMER.ko 2023-07-20T18:29:05.965 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_LED.ko 2023-07-20T18:29:05.965 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_LOG.ko 2023-07-20T18:29:05.966 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_MASQUERADE.ko 2023-07-20T18:29:05.966 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_NETMAP.ko 2023-07-20T18:29:05.966 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_NFLOG.ko 2023-07-20T18:29:05.966 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_NFQUEUE.ko 2023-07-20T18:29:05.966 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_RATEEST.ko 2023-07-20T18:29:05.966 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_REDIRECT.ko 2023-07-20T18:29:05.966 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_SECMARK.ko 2023-07-20T18:29:05.966 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_TCPMSS.ko 2023-07-20T18:29:05.967 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-07-20T18:29:05.967 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_TEE.ko 2023-07-20T18:29:05.967 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_TPROXY.ko 2023-07-20T18:29:05.967 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_TRACE.ko 2023-07-20T18:29:05.967 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_addrtype.ko 2023-07-20T18:29:05.967 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_bpf.ko 2023-07-20T18:29:05.967 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_cgroup.ko 2023-07-20T18:29:05.967 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_cluster.ko 2023-07-20T18:29:05.968 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_comment.ko 2023-07-20T18:29:05.968 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_connbytes.ko 2023-07-20T18:29:05.968 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_connlabel.ko 2023-07-20T18:29:05.968 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_connlimit.ko 2023-07-20T18:29:05.968 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_connmark.ko 2023-07-20T18:29:05.968 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_conntrack.ko 2023-07-20T18:29:05.968 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_cpu.ko 2023-07-20T18:29:05.968 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_dccp.ko 2023-07-20T18:29:05.969 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_devgroup.ko 2023-07-20T18:29:05.969 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_dscp.ko 2023-07-20T18:29:05.969 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_ecn.ko 2023-07-20T18:29:05.969 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_esp.ko 2023-07-20T18:29:05.969 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_hashlimit.ko 2023-07-20T18:29:05.969 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_helper.ko 2023-07-20T18:29:05.969 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_hl.ko 2023-07-20T18:29:05.970 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_ipcomp.ko 2023-07-20T18:29:05.970 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_iprange.ko 2023-07-20T18:29:05.970 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_ipvs.ko 2023-07-20T18:29:05.970 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_l2tp.ko 2023-07-20T18:29:05.970 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_length.ko 2023-07-20T18:29:05.970 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_limit.ko 2023-07-20T18:29:05.970 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_mac.ko 2023-07-20T18:29:05.970 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_mark.ko 2023-07-20T18:29:05.971 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_multiport.ko 2023-07-20T18:29:05.971 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_nat.ko 2023-07-20T18:29:05.971 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_nfacct.ko 2023-07-20T18:29:05.971 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_osf.ko 2023-07-20T18:29:05.971 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_owner.ko 2023-07-20T18:29:05.971 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_physdev.ko 2023-07-20T18:29:05.971 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_pkttype.ko 2023-07-20T18:29:05.971 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_policy.ko 2023-07-20T18:29:05.972 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_quota.ko 2023-07-20T18:29:05.972 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_rateest.ko 2023-07-20T18:29:05.972 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_realm.ko 2023-07-20T18:29:05.972 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_recent.ko 2023-07-20T18:29:05.972 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_sctp.ko 2023-07-20T18:29:05.972 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_set.ko 2023-07-20T18:29:05.972 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_socket.ko 2023-07-20T18:29:05.972 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_state.ko 2023-07-20T18:29:05.973 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_statistic.ko 2023-07-20T18:29:05.973 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_string.ko 2023-07-20T18:29:05.973 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_tcpmss.ko 2023-07-20T18:29:05.973 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_time.ko 2023-07-20T18:29:05.973 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netfilter/xt_u32.ko 2023-07-20T18:29:05.973 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netlink 2023-07-20T18:29:05.973 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/netlink/netlink_diag.ko 2023-07-20T18:29:05.974 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/nsh 2023-07-20T18:29:05.974 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/nsh/nsh.ko 2023-07-20T18:29:05.974 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/openvswitch 2023-07-20T18:29:05.974 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/openvswitch/openvswitch.ko 2023-07-20T18:29:05.974 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/openvswitch/vport-geneve.ko 2023-07-20T18:29:05.974 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/openvswitch/vport-gre.ko 2023-07-20T18:29:05.974 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/openvswitch/vport-vxlan.ko 2023-07-20T18:29:05.974 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/packet 2023-07-20T18:29:05.975 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/packet/af_packet_diag.ko 2023-07-20T18:29:05.975 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/rds 2023-07-20T18:29:05.975 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/rds/rds.ko 2023-07-20T18:29:05.975 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/rds/rds_rdma.ko 2023-07-20T18:29:05.975 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/rds/rds_tcp.ko 2023-07-20T18:29:05.975 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched 2023-07-20T18:29:05.975 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/act_csum.ko 2023-07-20T18:29:05.975 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/act_gact.ko 2023-07-20T18:29:05.976 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/act_ipt.ko 2023-07-20T18:29:05.976 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/act_mirred.ko 2023-07-20T18:29:05.976 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/act_nat.ko 2023-07-20T18:29:05.976 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/act_pedit.ko 2023-07-20T18:29:05.976 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/act_police.ko 2023-07-20T18:29:05.976 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/act_simple.ko 2023-07-20T18:29:05.976 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/act_skbedit.ko 2023-07-20T18:29:05.976 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/act_vlan.ko 2023-07-20T18:29:05.977 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/cls_basic.ko 2023-07-20T18:29:05.977 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/cls_bpf.ko 2023-07-20T18:29:05.977 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/cls_flow.ko 2023-07-20T18:29:05.977 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/cls_fw.ko 2023-07-20T18:29:05.977 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/cls_route.ko 2023-07-20T18:29:05.977 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/cls_u32.ko 2023-07-20T18:29:05.977 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/em_cmp.ko 2023-07-20T18:29:05.977 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/em_ipset.ko 2023-07-20T18:29:05.978 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/em_meta.ko 2023-07-20T18:29:05.978 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/em_nbyte.ko 2023-07-20T18:29:05.978 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/em_text.ko 2023-07-20T18:29:05.978 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/em_u32.ko 2023-07-20T18:29:05.978 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_choke.ko 2023-07-20T18:29:05.978 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_codel.ko 2023-07-20T18:29:05.978 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_drr.ko 2023-07-20T18:29:05.978 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_fq.ko 2023-07-20T18:29:05.979 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_gred.ko 2023-07-20T18:29:05.979 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_hfsc.ko 2023-07-20T18:29:05.979 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_hhf.ko 2023-07-20T18:29:05.979 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_htb.ko 2023-07-20T18:29:05.979 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_ingress.ko 2023-07-20T18:29:05.979 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_mqprio.ko 2023-07-20T18:29:05.979 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_mqprio_lib.ko 2023-07-20T18:29:05.980 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_multiq.ko 2023-07-20T18:29:05.980 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_netem.ko 2023-07-20T18:29:05.980 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_pie.ko 2023-07-20T18:29:05.980 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_plug.ko 2023-07-20T18:29:05.980 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_prio.ko 2023-07-20T18:29:05.980 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_qfq.ko 2023-07-20T18:29:05.980 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_red.ko 2023-07-20T18:29:05.980 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_sfb.ko 2023-07-20T18:29:05.981 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_sfq.ko 2023-07-20T18:29:05.981 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_tbf.ko 2023-07-20T18:29:05.981 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sched/sch_teql.ko 2023-07-20T18:29:05.981 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sctp 2023-07-20T18:29:05.981 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sctp/sctp.ko 2023-07-20T18:29:05.981 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sctp/sctp_diag.ko 2023-07-20T18:29:05.981 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sunrpc 2023-07-20T18:29:05.981 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sunrpc/auth_gss 2023-07-20T18:29:05.982 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-07-20T18:29:05.982 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-07-20T18:29:05.982 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sunrpc/sunrpc.ko 2023-07-20T18:29:05.982 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sunrpc/xprtrdma 2023-07-20T18:29:05.982 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-07-20T18:29:05.982 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/tipc 2023-07-20T18:29:05.982 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/tipc/diag.ko 2023-07-20T18:29:05.982 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/tipc/tipc.ko 2023-07-20T18:29:05.983 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/unix 2023-07-20T18:29:05.983 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/unix/unix_diag.ko 2023-07-20T18:29:05.983 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/vmw_vsock 2023-07-20T18:29:05.983 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-07-20T18:29:05.983 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-07-20T18:29:05.983 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/vmw_vsock/vsock.ko 2023-07-20T18:29:05.983 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/vmw_vsock/vsock_diag.ko 2023-07-20T18:29:05.983 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/vmw_vsock/vsock_loopback.ko 2023-07-20T18:29:05.984 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/xfrm 2023-07-20T18:29:05.984 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/net/xfrm/xfrm_ipcomp.ko 2023-07-20T18:29:05.984 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/security 2023-07-20T18:29:05.984 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/security/keys 2023-07-20T18:29:05.984 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/security/keys/encrypted-keys 2023-07-20T18:29:05.984 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-07-20T18:29:05.984 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/security/keys/trusted-keys 2023-07-20T18:29:05.984 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/security/keys/trusted-keys/trusted.ko 2023-07-20T18:29:05.985 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/virt 2023-07-20T18:29:05.985 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/virt/lib 2023-07-20T18:29:05.985 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/kernel/virt/lib/irqbypass.ko 2023-07-20T18:29:05.985 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.alias 2023-07-20T18:29:05.985 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.alias.bin 2023-07-20T18:29:05.985 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.builtin 2023-07-20T18:29:05.985 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.builtin.bin 2023-07-20T18:29:05.986 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.builtin.modinfo 2023-07-20T18:29:05.986 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.dep 2023-07-20T18:29:05.986 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.dep.bin 2023-07-20T18:29:05.986 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.devname 2023-07-20T18:29:05.986 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.order 2023-07-20T18:29:05.986 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.softdep 2023-07-20T18:29:05.986 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.symbols 2023-07-20T18:29:05.986 INFO:teuthology.orchestra.run.smithi057.stdout:/lib/modules/6.5.0-rc1-gc685da6f56cc/modules.symbols.bin 2023-07-20T18:29:05.987 DEBUG:teuthology.task.kernel:get_image_version: 6.5.0-rc1-gc685da6f56cc 2023-07-20T18:29:05.988 DEBUG:teuthology.orchestra.run.smithi057:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-07-20T18:29:06.016 INFO:teuthology.orchestra.run.smithi057.stdout:postinstall scriptlet (using /bin/sh): 2023-07-20T18:29:06.017 INFO:teuthology.orchestra.run.smithi057.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.5.0-rc1-gc685da6f56cc -a -r /boot/System.map-6.5.0-rc1-gc685da6f56cc ]; then 2023-07-20T18:29:06.017 INFO:teuthology.orchestra.run.smithi057.stdout:cp /boot/vmlinuz-6.5.0-rc1-gc685da6f56cc /boot/.vmlinuz-6.5.0-rc1-gc685da6f56cc-rpm 2023-07-20T18:29:06.017 INFO:teuthology.orchestra.run.smithi057.stdout:cp /boot/System.map-6.5.0-rc1-gc685da6f56cc /boot/.System.map-6.5.0-rc1-gc685da6f56cc-rpm 2023-07-20T18:29:06.017 INFO:teuthology.orchestra.run.smithi057.stdout:rm -f /boot/vmlinuz-6.5.0-rc1-gc685da6f56cc /boot/System.map-6.5.0-rc1-gc685da6f56cc 2023-07-20T18:29:06.017 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/installkernel 6.5.0-rc1-gc685da6f56cc /boot/.vmlinuz-6.5.0-rc1-gc685da6f56cc-rpm /boot/.System.map-6.5.0-rc1-gc685da6f56cc-rpm 2023-07-20T18:29:06.017 INFO:teuthology.orchestra.run.smithi057.stdout:rm -f /boot/.vmlinuz-6.5.0-rc1-gc685da6f56cc-rpm /boot/.System.map-6.5.0-rc1-gc685da6f56cc-rpm 2023-07-20T18:29:06.017 INFO:teuthology.orchestra.run.smithi057.stdout:fi 2023-07-20T18:29:06.018 INFO:teuthology.orchestra.run.smithi057.stdout:preuninstall scriptlet (using /bin/sh): 2023-07-20T18:29:06.018 INFO:teuthology.orchestra.run.smithi057.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-07-20T18:29:06.018 INFO:teuthology.orchestra.run.smithi057.stdout:new-kernel-pkg --remove 6.5.0-rc1-gc685da6f56cc --rminitrd --initrdfile=/boot/initramfs-6.5.0-rc1-gc685da6f56cc.img 2023-07-20T18:29:06.018 INFO:teuthology.orchestra.run.smithi057.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-07-20T18:29:06.018 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-install remove 6.5.0-rc1-gc685da6f56cc 2023-07-20T18:29:06.018 INFO:teuthology.orchestra.run.smithi057.stdout:fi 2023-07-20T18:29:06.018 INFO:teuthology.orchestra.run.smithi057.stdout:postuninstall scriptlet (using /bin/sh): 2023-07-20T18:29:06.019 INFO:teuthology.orchestra.run.smithi057.stdout:if [ -x /sbin/update-bootloader ]; then 2023-07-20T18:29:06.019 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/update-bootloader --remove 6.5.0-rc1-gc685da6f56cc 2023-07-20T18:29:06.019 INFO:teuthology.orchestra.run.smithi057.stdout:fi 2023-07-20T18:29:06.020 DEBUG:teuthology.orchestra.run.smithi057:> sudo rpm -qi grub2-tools 2023-07-20T18:29:06.100 INFO:teuthology.orchestra.run.smithi057.stdout:Name : grub2-tools 2023-07-20T18:29:06.100 INFO:teuthology.orchestra.run.smithi057.stdout:Epoch : 1 2023-07-20T18:29:06.101 INFO:teuthology.orchestra.run.smithi057.stdout:Version : 2.02 2023-07-20T18:29:06.101 INFO:teuthology.orchestra.run.smithi057.stdout:Release : 123.el8 2023-07-20T18:29:06.101 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: x86_64 2023-07-20T18:29:06.102 INFO:teuthology.orchestra.run.smithi057.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2023-07-20T18:29:06.102 INFO:teuthology.orchestra.run.smithi057.stdout:Group : System Environment/Base 2023-07-20T18:29:06.102 INFO:teuthology.orchestra.run.smithi057.stdout:Size : 9265450 2023-07-20T18:29:06.102 INFO:teuthology.orchestra.run.smithi057.stdout:License : GPLv3+ 2023-07-20T18:29:06.103 INFO:teuthology.orchestra.run.smithi057.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2023-07-20T18:29:06.103 INFO:teuthology.orchestra.run.smithi057.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2023-07-20T18:29:06.103 INFO:teuthology.orchestra.run.smithi057.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2023-07-20T18:29:06.104 INFO:teuthology.orchestra.run.smithi057.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2023-07-20T18:29:06.104 INFO:teuthology.orchestra.run.smithi057.stdout:Relocations : (not relocatable) 2023-07-20T18:29:06.104 INFO:teuthology.orchestra.run.smithi057.stdout:Packager : Red Hat, Inc. 2023-07-20T18:29:06.104 INFO:teuthology.orchestra.run.smithi057.stdout:Vendor : Red Hat, Inc. 2023-07-20T18:29:06.105 INFO:teuthology.orchestra.run.smithi057.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-20T18:29:06.105 INFO:teuthology.orchestra.run.smithi057.stdout:Summary : Support tools for GRUB. 2023-07-20T18:29:06.105 INFO:teuthology.orchestra.run.smithi057.stdout:Description : 2023-07-20T18:29:06.106 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:29:06.106 INFO:teuthology.orchestra.run.smithi057.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-20T18:29:06.107 INFO:teuthology.orchestra.run.smithi057.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-20T18:29:06.107 INFO:teuthology.orchestra.run.smithi057.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-20T18:29:06.107 INFO:teuthology.orchestra.run.smithi057.stdout:hardware devices. 2023-07-20T18:29:06.108 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:29:06.108 INFO:teuthology.orchestra.run.smithi057.stdout:This subpackage provides tools for support of all platforms. 2023-07-20T18:29:06.110 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-20T18:29:06.110 INFO:teuthology.task.kernel:Updating grub on smithi057 to boot 6.5.0-rc1-gc685da6f56cc 2023-07-20T18:29:06.110 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-20T18:29:08.142 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2023-07-20T18:29:09.778 INFO:teuthology.orchestra.run.smithi057.stderr:done 2023-07-20T18:29:09.780 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2023-07-20T18:29:09.800 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.zQtkYDabwE 2023-07-20T18:29:09.800 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zQtkYDabwE 2023-07-20T18:29:09.870 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.zQtkYDabwE 2023-07-20T18:29:10.014 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.zQtkYDabwE is 6KB 2023-07-20T18:29:10.063 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.zQtkYDabwE 2023-07-20T18:29:10.080 DEBUG:teuthology.orchestra.run.smithi057:> sudo /bin/ls /boot/loader/entries || true 2023-07-20T18:29:10.145 INFO:teuthology.orchestra.run.smithi057.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2023-07-20T18:29:10.145 INFO:teuthology.orchestra.run.smithi057.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2023-07-20T18:29:10.146 INFO:teuthology.orchestra.run.smithi057.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.5.0-rc1-gc685da6f56cc.conf 2023-07-20T18:29:10.146 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.5.0-rc1-gc685da6f56cc 2023-07-20T18:29:10.301 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2023-07-20T18:29:40.306 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-07-20T18:29:40.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-07-20T18:29:40.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:29:58.768 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-07-20T18:30:07.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-07-20T18:30:07.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:30:10.832 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-07-20T18:30:22.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-07-20T18:30:22.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:31:22.837 DEBUG:teuthology.orchestra.remote:timed out 2023-07-20T18:31:37.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-07-20T18:31:37.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:31:38.189 DEBUG:teuthology.orchestra.run.smithi057:> true 2023-07-20T18:31:38.599 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-07-20T18:31:38.600 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.5.0-rc1-gc685da6f56cc"... 2023-07-20T18:31:38.600 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2023-07-20T18:31:38.643 INFO:teuthology.orchestra.run.smithi057.stdout:6.5.0-rc1-gc685da6f56cc 2023-07-20T18:31:38.643 DEBUG:teuthology.task.kernel:current kernel version is 6.5.0-rc1-gc685da6f56cc vs 6.5.0-rc1-gc685da6f56cc 2023-07-20T18:31:38.644 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-20T18:31:38.644 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-07-20T18:31:39.645 DEBUG:teuthology.parallel:result is None 2023-07-20T18:31:39.646 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-20T18:31:39.653 INFO:teuthology.task.internal:Creating test directory... 2023-07-20T18:31:39.654 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T18:31:39.661 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T18:31:39.666 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T18:31:39.703 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-20T18:31:39.714 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-20T18:31:39.721 INFO:teuthology.task.internal:Creating archive directory... 2023-07-20T18:31:39.722 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T18:31:39.749 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T18:31:39.752 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T18:31:39.809 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-20T18:31:39.817 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-20T18:31:39.818 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T18:31:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:31:39.845 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T18:31:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:31:39.864 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T18:31:39.869 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T18:31:39.888 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T18:31:39.937 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:31:39.950 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:31:39.958 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:31:39.974 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:31:39.982 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:31:40.018 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:31:40.021 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-20T18:31:40.029 INFO:teuthology.task.internal:Configuring sudo... 2023-07-20T18:31:40.029 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T18:31:40.034 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T18:31:40.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T18:31:40.100 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-20T18:31:40.265 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-20T18:31:40.266 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T18:31:40.271 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T18:31:40.273 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T18:31:40.309 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:31:40.425 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:31:40.505 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:31:40.581 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:31:40.661 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:31:40.661 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T18:31:40.729 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:31:40.783 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:31:40.834 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:31:40.924 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:31:40.965 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:31:40.965 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T18:31:41.034 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:31:41.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:31:41.140 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:31:41.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:31:41.268 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:31:41.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T18:31:41.337 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2023-07-20T18:31:41.342 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2023-07-20T18:31:41.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2023-07-20T18:31:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T18:31:41.438 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T18:31:41.464 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T18:31:41.845 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-20T18:31:41.857 INFO:teuthology.task.internal:Starting timer... 2023-07-20T18:31:41.858 INFO:teuthology.run_tasks:Running task pcp... 2023-07-20T18:31:41.869 INFO:teuthology.run_tasks:Running task selinux... 2023-07-20T18:31:41.881 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-20T18:31:41.882 DEBUG:teuthology.orchestra.run.smithi057:> sudo service auditd rotate 2023-07-20T18:31:42.005 INFO:teuthology.orchestra.run.smithi057.stdout:Rotating logs: 2023-07-20T18:31:42.009 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2023-07-20T18:31:42.067 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2023-07-20T18:31:42.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2023-07-20T18:31:42.135 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2023-07-20T18:31:42.137 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-20T18:31:42.137 DEBUG:teuthology.orchestra.run.smithi057:> /usr/sbin/getenforce 2023-07-20T18:31:42.190 INFO:teuthology.orchestra.run.smithi057.stdout:Permissive 2023-07-20T18:31:42.191 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2023-07-20T18:31:42.218 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2023-07-20T18:31:42.219 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2023-07-20T18:31:42.243 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2023-07-20T18:31:42.243 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi057.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2023-07-20T18:31:42.244 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T18:31:42.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:31:42.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T18:31:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:31:42.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T18:31:42.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:31:42.374 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-20T18:31:42.374 DEBUG:teuthology.orchestra.run.smithi057:> sudo /usr/sbin/setenforce permissive 2023-07-20T18:31:42.446 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2023-07-20T18:31:42.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2023-07-20T18:31:42.516 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-20T18:31:42.530 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-20T18:31:42.536 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-20T18:31:42.724 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-20T18:31:42.744 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-20T18:31:42.746 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi057.front.sepia.ceph.com,smithi088.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-20T18:45:43.939 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi057.front.sepia.ceph.com'), Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2023-07-20T18:45:43.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-07-20T18:45:43.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:45:44.032 DEBUG:teuthology.orchestra.run.smithi057:> true 2023-07-20T18:45:44.138 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-07-20T18:45:44.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-07-20T18:45:44.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:45:44.207 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-07-20T18:45:44.275 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-07-20T18:45:44.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-07-20T18:45:44.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:45:44.346 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-07-20T18:45:44.415 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-07-20T18:45:44.416 INFO:teuthology.run_tasks:Running task clock... 2023-07-20T18:45:44.427 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-20T18:45:44.427 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T18:45:44.428 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T18:45:44.432 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T18:45:44.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T18:45:44.435 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T18:45:44.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T18:45:44.468 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T18:45:44.484 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T18:45:44.492 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T18:45:44.506 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T18:45:44.508 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T18:45:44.542 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T18:45:44.563 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2023-07-20T18:45:44.576 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2023-07-20T18:45:44.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2023-07-20T18:45:44.591 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T18:45:44.599 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2023-07-20T18:45:44.605 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T18:45:44.614 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T18:45:44.628 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T18:45:44.639 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: ntpd: command not found 2023-07-20T18:45:44.669 INFO:teuthology.orchestra.run.smithi057.stdout:506 Cannot talk to daemon 2023-07-20T18:45:44.692 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2023-07-20T18:45:44.696 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T18:45:44.696 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2023-07-20T18:45:44.696 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.696 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.696 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.697 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.704 INFO:teuthology.orchestra.run.smithi192.stderr:bash: ntpq: command not found 2023-07-20T18:45:44.705 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T18:45:44.708 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T18:45:44.708 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2023-07-20T18:45:44.708 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.709 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.709 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.709 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.740 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T18:45:44.832 INFO:teuthology.orchestra.run.smithi057.stderr:bash: ntpq: command not found 2023-07-20T18:45:44.841 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T18:45:44.841 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2023-07-20T18:45:44.841 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.841 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.841 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.842 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:45:44.843 INFO:teuthology.run_tasks:Running task pexec... 2023-07-20T18:45:44.862 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-20T18:45:44.863 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi057.front.sepia.ceph.com 2023-07-20T18:45:44.864 DEBUG:teuthology.orchestra.run.smithi057:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T18:45:44.864 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi088.front.sepia.ceph.com 2023-07-20T18:45:44.864 DEBUG:teuthology.orchestra.run.smithi088:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T18:45:44.865 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi192.front.sepia.ceph.com 2023-07-20T18:45:44.865 DEBUG:teuthology.orchestra.run.smithi192:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T18:45:45.243 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-20T18:45:45.249 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-07-20T18:45:45.392 INFO:teuthology.orchestra.run.smithi057.stdout:Updating Subscription Management repositories. 2023-07-20T18:45:46.868 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:34 ago on Thu 20 Jul 2023 06:40:12 PM UTC. 2023-07-20T18:45:46.909 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:05:34 ago on Thu 20 Jul 2023 06:40:12 PM UTC. 2023-07-20T18:45:47.116 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:18 ago on Thu 20 Jul 2023 06:43:28 PM UTC. 2023-07-20T18:45:48.603 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-07-20T18:45:48.604 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-07-20T18:45:48.604 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2023-07-20T18:45:48.604 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-07-20T18:45:48.604 INFO:teuthology.orchestra.run.smithi192.stdout:Resetting modules: 2023-07-20T18:45:48.604 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools 2023-07-20T18:45:48.605 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:45:48.605 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-07-20T18:45:48.605 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-07-20T18:45:48.605 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:45:48.634 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-20T18:45:48.635 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-07-20T18:45:48.635 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2023-07-20T18:45:48.635 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-07-20T18:45:48.635 INFO:teuthology.orchestra.run.smithi088.stdout:Resetting modules: 2023-07-20T18:45:48.636 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools 2023-07-20T18:45:48.636 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:45:48.636 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-07-20T18:45:48.636 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-07-20T18:45:48.636 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:45:49.212 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-07-20T18:45:49.214 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-07-20T18:45:49.214 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2023-07-20T18:45:49.214 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-07-20T18:45:49.214 INFO:teuthology.orchestra.run.smithi057.stdout:Resetting modules: 2023-07-20T18:45:49.214 INFO:teuthology.orchestra.run.smithi057.stdout: container-tools 2023-07-20T18:45:49.214 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:45:49.215 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-07-20T18:45:49.215 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-07-20T18:45:49.215 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:45:49.568 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:37 ago on Thu 20 Jul 2023 06:40:12 PM UTC. 2023-07-20T18:45:49.586 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:05:37 ago on Thu 20 Jul 2023 06:40:12 PM UTC. 2023-07-20T18:45:50.477 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:21 ago on Thu 20 Jul 2023 06:43:28 PM UTC. 2023-07-20T18:45:57.646 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-07-20T18:45:57.711 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-20T18:45:58.107 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-07-20T18:45:58.189 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-20T18:45:59.953 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:47 ago on Thu 20 Jul 2023 06:40:12 PM UTC. 2023-07-20T18:46:00.108 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:05:47 ago on Thu 20 Jul 2023 06:40:12 PM UTC. 2023-07-20T18:46:01.122 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-07-20T18:46:01.744 INFO:teuthology.orchestra.run.smithi057.stdout:Updating Subscription Management repositories. 2023-07-20T18:46:01.816 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-07-20T18:46:01.829 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-20T18:46:01.910 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================================ 2023-07-20T18:46:01.910 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-07-20T18:46:01.911 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================================ 2023-07-20T18:46:01.912 INFO:teuthology.orchestra.run.smithi192.stdout:Installing group/module packages: 2023-07-20T18:46:01.912 INFO:teuthology.orchestra.run.smithi192.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T18:46:01.912 INFO:teuthology.orchestra.run.smithi192.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T18:46:01.913 INFO:teuthology.orchestra.run.smithi192.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T18:46:01.913 INFO:teuthology.orchestra.run.smithi192.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T18:46:01.913 INFO:teuthology.orchestra.run.smithi192.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T18:46:01.913 INFO:teuthology.orchestra.run.smithi192.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T18:46:01.913 INFO:teuthology.orchestra.run.smithi192.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T18:46:01.913 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-07-20T18:46:01.913 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T18:46:01.914 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T18:46:01.914 INFO:teuthology.orchestra.run.smithi192.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T18:46:01.914 INFO:teuthology.orchestra.run.smithi192.stdout:Installing module profiles: 2023-07-20T18:46:01.914 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools/common 2023-07-20T18:46:01.914 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:46:01.914 INFO:teuthology.orchestra.run.smithi192.stdout:Enabling module streams: 2023-07-20T18:46:01.914 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools rhel8 2023-07-20T18:46:01.915 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:46:01.915 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-07-20T18:46:01.915 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================================ 2023-07-20T18:46:01.915 INFO:teuthology.orchestra.run.smithi192.stdout:Install 10 Packages 2023-07-20T18:46:01.915 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:46:01.916 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 18 M 2023-07-20T18:46:01.916 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 65 M 2023-07-20T18:46:01.916 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-07-20T18:46:01.919 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================================ 2023-07-20T18:46:01.920 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-07-20T18:46:01.920 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================================ 2023-07-20T18:46:01.920 INFO:teuthology.orchestra.run.smithi088.stdout:Installing group/module packages: 2023-07-20T18:46:01.920 INFO:teuthology.orchestra.run.smithi088.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T18:46:01.920 INFO:teuthology.orchestra.run.smithi088.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T18:46:01.920 INFO:teuthology.orchestra.run.smithi088.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T18:46:01.920 INFO:teuthology.orchestra.run.smithi088.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T18:46:01.920 INFO:teuthology.orchestra.run.smithi088.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T18:46:01.921 INFO:teuthology.orchestra.run.smithi088.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T18:46:01.921 INFO:teuthology.orchestra.run.smithi088.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T18:46:01.921 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-07-20T18:46:01.921 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T18:46:01.921 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T18:46:01.921 INFO:teuthology.orchestra.run.smithi088.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T18:46:01.921 INFO:teuthology.orchestra.run.smithi088.stdout:Installing module profiles: 2023-07-20T18:46:01.921 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools/common 2023-07-20T18:46:01.922 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:46:01.922 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling module streams: 2023-07-20T18:46:01.922 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools rhel8 2023-07-20T18:46:01.922 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:46:01.922 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-07-20T18:46:01.922 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================================ 2023-07-20T18:46:01.922 INFO:teuthology.orchestra.run.smithi088.stdout:Install 10 Packages 2023-07-20T18:46:01.923 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:46:01.923 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 18 M 2023-07-20T18:46:01.923 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 65 M 2023-07-20T18:46:01.923 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-07-20T18:46:02.054 INFO:teuthology.orchestra.run.smithi192.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 175 kB/s | 25 kB 00:00 2023-07-20T18:46:02.101 INFO:teuthology.orchestra.run.smithi088.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 138 kB/s | 25 kB 00:00 2023-07-20T18:46:02.127 INFO:teuthology.orchestra.run.smithi088.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 199 kB/s | 41 kB 00:00 2023-07-20T18:46:02.129 INFO:teuthology.orchestra.run.smithi192.stdout:(2/10): python3-pyxdg-0.25-16.el8.noarch.rpm 431 kB/s | 94 kB 00:00 2023-07-20T18:46:02.180 INFO:teuthology.orchestra.run.smithi192.stdout:(3/10): udica-0.2.6-4.module+el8.6.0+16771+28df 395 kB/s | 49 kB 00:00 2023-07-20T18:46:02.194 INFO:teuthology.orchestra.run.smithi088.stdout:(3/10): udica-0.2.6-4.module+el8.6.0+16771+28df 537 kB/s | 49 kB 00:00 2023-07-20T18:46:02.205 INFO:teuthology.orchestra.run.smithi192.stdout:(4/10): yajl-2.1.0-10.el8.x86_64.rpm 139 kB/s | 41 kB 00:00 2023-07-20T18:46:02.227 INFO:teuthology.orchestra.run.smithi088.stdout:(4/10): python3-pyxdg-0.25-16.el8.noarch.rpm 308 kB/s | 94 kB 00:00 2023-07-20T18:46:02.263 INFO:teuthology.orchestra.run.smithi192.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.8 MB/s | 524 kB 00:00 2023-07-20T18:46:02.328 INFO:teuthology.orchestra.run.smithi088.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.6 MB/s | 524 kB 00:00 2023-07-20T18:46:02.330 INFO:teuthology.orchestra.run.smithi192.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.2 MB/s | 149 kB 00:00 2023-07-20T18:46:02.353 INFO:teuthology.orchestra.run.smithi088.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.2 MB/s | 149 kB 00:00 2023-07-20T18:46:02.437 INFO:teuthology.orchestra.run.smithi088.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.9 MB/s | 213 kB 00:00 2023-07-20T18:46:02.538 INFO:teuthology.orchestra.run.smithi088.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 23 MB/s | 8.0 MB 00:00 2023-07-20T18:46:02.612 INFO:teuthology.orchestra.run.smithi088.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 13 MB/s | 2.2 MB 00:00 2023-07-20T18:46:02.689 INFO:teuthology.orchestra.run.smithi192.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 16 MB/s | 8.0 MB 00:00 2023-07-20T18:46:02.714 INFO:teuthology.orchestra.run.smithi192.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 472 kB/s | 213 kB 00:00 2023-07-20T18:46:02.755 INFO:teuthology.orchestra.run.smithi088.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 18 MB/s | 7.0 MB 00:00 2023-07-20T18:46:02.757 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:46:02.758 INFO:teuthology.orchestra.run.smithi088.stdout:Total 22 MB/s | 18 MB 00:00 2023-07-20T18:46:02.848 INFO:teuthology.orchestra.run.smithi192.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 14 MB/s | 7.0 MB 00:00 2023-07-20T18:46:02.899 INFO:teuthology.orchestra.run.smithi192.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 11 MB/s | 2.2 MB 00:00 2023-07-20T18:46:02.899 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:46:02.899 INFO:teuthology.orchestra.run.smithi192.stdout:Total 18 MB/s | 18 MB 00:00 2023-07-20T18:46:02.953 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-07-20T18:46:02.972 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-07-20T18:46:02.972 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-07-20T18:46:03.100 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-07-20T18:46:03.118 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-07-20T18:46:03.119 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-07-20T18:46:03.156 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-07-20T18:46:03.160 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-07-20T18:46:03.300 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-07-20T18:46:03.304 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-07-20T18:46:03.546 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:34 ago on Thu 20 Jul 2023 06:43:28 PM UTC. 2023-07-20T18:46:03.656 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-07-20T18:46:03.806 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-07-20T18:46:03.817 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T18:46:03.978 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:46:03.982 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T18:46:04.137 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:46:04.163 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T18:46:04.364 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T18:46:04.388 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T18:46:04.567 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T18:46:04.731 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T18:46:04.932 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T18:46:05.489 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-07-20T18:46:05.489 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T18:46:05.594 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================================ 2023-07-20T18:46:05.594 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-07-20T18:46:05.594 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================================ 2023-07-20T18:46:05.594 INFO:teuthology.orchestra.run.smithi057.stdout:Installing group/module packages: 2023-07-20T18:46:05.595 INFO:teuthology.orchestra.run.smithi057.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T18:46:05.595 INFO:teuthology.orchestra.run.smithi057.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T18:46:05.595 INFO:teuthology.orchestra.run.smithi057.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T18:46:05.595 INFO:teuthology.orchestra.run.smithi057.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T18:46:05.596 INFO:teuthology.orchestra.run.smithi057.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T18:46:05.596 INFO:teuthology.orchestra.run.smithi057.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T18:46:05.596 INFO:teuthology.orchestra.run.smithi057.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T18:46:05.596 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-07-20T18:46:05.597 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T18:46:05.597 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T18:46:05.597 INFO:teuthology.orchestra.run.smithi057.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T18:46:05.597 INFO:teuthology.orchestra.run.smithi057.stdout:Installing module profiles: 2023-07-20T18:46:05.597 INFO:teuthology.orchestra.run.smithi057.stdout: container-tools/common 2023-07-20T18:46:05.598 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:46:05.598 INFO:teuthology.orchestra.run.smithi057.stdout:Enabling module streams: 2023-07-20T18:46:05.598 INFO:teuthology.orchestra.run.smithi057.stdout: container-tools rhel8 2023-07-20T18:46:05.598 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:46:05.599 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-07-20T18:46:05.599 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================================ 2023-07-20T18:46:05.599 INFO:teuthology.orchestra.run.smithi057.stdout:Install 10 Packages 2023-07-20T18:46:05.599 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:46:05.601 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 18 M 2023-07-20T18:46:05.601 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 65 M 2023-07-20T18:46:05.602 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-07-20T18:46:05.686 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T18:46:05.716 INFO:teuthology.orchestra.run.smithi057.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 790 kB/s | 94 kB 00:00 2023-07-20T18:46:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 283 kB/s | 41 kB 00:00 2023-07-20T18:46:05.775 INFO:teuthology.orchestra.run.smithi057.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 140 kB/s | 25 kB 00:00 2023-07-20T18:46:05.850 INFO:teuthology.orchestra.run.smithi057.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 369 kB/s | 49 kB 00:00 2023-07-20T18:46:05.884 INFO:teuthology.orchestra.run.smithi057.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.6 MB/s | 524 kB 00:00 2023-07-20T18:46:05.917 INFO:teuthology.orchestra.run.smithi057.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.2 MB/s | 149 kB 00:00 2023-07-20T18:46:06.068 INFO:teuthology.orchestra.run.smithi057.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.1 MB/s | 213 kB 00:00 2023-07-20T18:46:06.176 INFO:teuthology.orchestra.run.smithi057.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 20 MB/s | 8.0 MB 00:00 2023-07-20T18:46:06.310 INFO:teuthology.orchestra.run.smithi057.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 9.1 MB/s | 2.2 MB 00:00 2023-07-20T18:46:06.391 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T18:46:06.410 INFO:teuthology.orchestra.run.smithi057.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 14 MB/s | 7.0 MB 00:00 2023-07-20T18:46:06.411 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:46:06.412 INFO:teuthology.orchestra.run.smithi057.stdout:Total 22 MB/s | 18 MB 00:00 2023-07-20T18:46:06.551 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T18:46:06.600 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T18:46:06.751 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-07-20T18:46:06.776 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-07-20T18:46:06.776 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-07-20T18:46:06.777 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T18:46:06.794 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T18:46:06.943 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T18:46:06.979 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:46:07.121 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:46:07.241 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:46:07.242 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T18:46:07.242 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:46:07.242 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T18:46:07.242 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T18:46:07.242 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T18:46:07.242 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T18:46:07.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T18:46:07.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T18:46:07.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T18:46:07.311 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-07-20T18:46:07.316 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-07-20T18:46:07.421 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:46:07.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T18:46:07.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:46:07.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T18:46:07.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T18:46:07.422 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T18:46:07.422 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T18:46:07.422 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T18:46:07.422 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T18:46:07.422 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T18:46:07.895 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-07-20T18:46:08.115 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T18:46:08.301 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:46:08.373 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T18:46:08.374 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:05:55 ago on Thu 20 Jul 2023 06:40:12 PM UTC. 2023-07-20T18:46:08.521 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T18:46:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T18:46:08.681 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:56 ago on Thu 20 Jul 2023 06:40:12 PM UTC. 2023-07-20T18:46:08.737 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T18:46:09.115 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T18:46:09.959 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T18:46:10.924 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T18:46:11.214 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T18:46:11.406 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T18:46:11.604 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:46:11.985 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:46:11.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T18:46:11.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:46:11.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T18:46:11.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T18:46:11.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T18:46:11.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T18:46:11.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T18:46:11.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T18:46:11.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T18:46:13.425 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T18:46:13.425 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:44 ago on Thu 20 Jul 2023 06:43:28 PM UTC. 2023-07-20T18:46:20.019 INFO:teuthology.orchestra.run.smithi088.stdout:Installed products updated. 2023-07-20T18:46:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:46:20.241 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-07-20T18:46:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:46:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:20.241 INFO:teuthology.orchestra.run.smithi088.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T18:46:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T18:46:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T18:46:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:46:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T18:46:20.242 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:46:20.243 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-20T18:46:20.403 DEBUG:teuthology.parallel:result is None 2023-07-20T18:46:20.696 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-07-20T18:46:20.926 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:46:20.926 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-07-20T18:46:20.927 INFO:teuthology.orchestra.run.smithi192.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:20.927 INFO:teuthology.orchestra.run.smithi192.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:46:20.927 INFO:teuthology.orchestra.run.smithi192.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:20.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T18:46:20.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T18:46:20.928 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T18:46:20.928 INFO:teuthology.orchestra.run.smithi192.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:20.928 INFO:teuthology.orchestra.run.smithi192.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:20.928 INFO:teuthology.orchestra.run.smithi192.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:46:20.928 INFO:teuthology.orchestra.run.smithi192.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T18:46:20.929 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:46:20.929 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-07-20T18:46:21.129 DEBUG:teuthology.parallel:result is None 2023-07-20T18:46:23.987 INFO:teuthology.orchestra.run.smithi057.stdout:Installed products updated. 2023-07-20T18:46:24.214 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:46:24.214 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-07-20T18:46:24.214 INFO:teuthology.orchestra.run.smithi057.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:24.214 INFO:teuthology.orchestra.run.smithi057.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:46:24.214 INFO:teuthology.orchestra.run.smithi057.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:24.215 INFO:teuthology.orchestra.run.smithi057.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T18:46:24.215 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T18:46:24.215 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T18:46:24.215 INFO:teuthology.orchestra.run.smithi057.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:24.215 INFO:teuthology.orchestra.run.smithi057.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:46:24.215 INFO:teuthology.orchestra.run.smithi057.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:46:24.215 INFO:teuthology.orchestra.run.smithi057.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T18:46:24.216 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:46:24.216 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-07-20T18:46:24.437 DEBUG:teuthology.parallel:result is None 2023-07-20T18:46:24.437 INFO:teuthology.run_tasks:Running task install... 2023-07-20T18:46:24.446 DEBUG:teuthology.task.install:project ceph 2023-07-20T18:46:24.446 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'}} 2023-07-20T18:46:24.447 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'} 2023-07-20T18:46:24.447 INFO:teuthology.task.install:Using flavor: default 2023-07-20T18:46:24.453 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-20T18:46:24.453 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-20T18:46:24.454 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T18:46:24.455 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:46:24.457 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T18:46:24.457 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:46:24.458 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T18:46:24.459 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:46:49.987 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T18:46:49.988 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T18:46:55.001 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T18:46:55.002 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T18:46:55.004 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T18:46:55.004 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T18:47:05.423 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T18:47:05.423 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:47:05.423 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T18:47:05.427 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T18:47:05.427 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:47:05.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T18:47:05.463 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T18:47:05.463 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T18:47:05.503 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T18:47:05.503 DEBUG:teuthology.orchestra.run.smithi057:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T18:47:05.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T18:47:05.538 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T18:47:05.539 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:47:05.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T18:47:05.574 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T18:47:05.574 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T18:47:05.615 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T18:47:05.617 DEBUG:teuthology.orchestra.run.smithi057:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T18:47:05.647 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T18:47:05.683 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2023-07-20T18:47:05.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2023-07-20T18:47:05.716 DEBUG:teuthology.orchestra.run.smithi057:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T18:47:05.728 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T18:47:05.784 INFO:teuthology.orchestra.run.smithi057.stdout:check_obsoletes = 1 2023-07-20T18:47:05.788 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum clean all 2023-07-20T18:47:05.796 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2023-07-20T18:47:05.797 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2023-07-20T18:47:06.058 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-07-20T18:47:06.169 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-20T18:47:06.269 INFO:teuthology.orchestra.run.smithi057.stdout:Updating Subscription Management repositories. 2023-07-20T18:47:07.162 INFO:teuthology.orchestra.run.smithi192.stdout:44 files removed 2023-07-20T18:47:07.184 INFO:teuthology.orchestra.run.smithi088.stdout:44 files removed 2023-07-20T18:47:07.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T18:47:07.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T18:47:07.306 INFO:teuthology.orchestra.run.smithi057.stdout:44 files removed 2023-07-20T18:47:07.363 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T18:47:07.552 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-07-20T18:47:07.570 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-20T18:47:07.843 INFO:teuthology.orchestra.run.smithi057.stdout:Updating Subscription Management repositories. 2023-07-20T18:47:09.839 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 74 kB/s | 81 kB 00:01 2023-07-20T18:47:10.034 INFO:teuthology.orchestra.run.smithi057.stdout:ceph packages for x86_64 82 kB/s | 81 kB 00:00 2023-07-20T18:47:10.106 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 81 kB/s | 81 kB 00:00 2023-07-20T18:47:10.524 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-07-20T18:47:10.752 INFO:teuthology.orchestra.run.smithi057.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-07-20T18:47:10.807 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-07-20T18:47:11.290 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 2.0 kB/s | 1.5 kB 00:00 2023-07-20T18:47:11.511 INFO:teuthology.orchestra.run.smithi057.stdout:ceph source packages 2.0 kB/s | 1.5 kB 00:00 2023-07-20T18:47:11.565 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 2.0 kB/s | 1.5 kB 00:00 2023-07-20T18:47:11.696 INFO:teuthology.orchestra.run.smithi192.stdout:Copr repo for python3-asyncssh owned by ceph 9.1 kB/s | 3.5 kB 00:00 2023-07-20T18:47:11.928 INFO:teuthology.orchestra.run.smithi057.stdout:Copr repo for python3-asyncssh owned by ceph 8.8 kB/s | 3.5 kB 00:00 2023-07-20T18:47:11.999 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 8.4 kB/s | 3.5 kB 00:00 2023-07-20T18:47:13.883 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 7.2 MB/s | 16 MB 00:02 2023-07-20T18:47:14.552 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 16 MB 00:02 2023-07-20T18:47:15.581 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 16 MB 00:03 2023-07-20T18:47:17.333 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 398 kB/s | 24 kB 00:00 2023-07-20T18:47:17.966 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 395 kB/s | 24 kB 00:00 2023-07-20T18:47:18.544 INFO:teuthology.orchestra.run.smithi192.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 47 MB/s | 53 MB 00:01 2023-07-20T18:47:19.193 INFO:teuthology.orchestra.run.smithi088.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 46 MB/s | 53 MB 00:01 2023-07-20T18:47:19.837 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 383 kB/s | 24 kB 00:00 2023-07-20T18:47:21.314 INFO:teuthology.orchestra.run.smithi057.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 38 MB/s | 53 MB 00:01 2023-07-20T18:47:26.928 INFO:teuthology.orchestra.run.smithi192.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 47 MB 00:01 2023-07-20T18:47:27.586 INFO:teuthology.orchestra.run.smithi088.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 47 MB 00:01 2023-07-20T18:47:30.565 INFO:teuthology.orchestra.run.smithi057.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 MB/s | 47 MB 00:01 2023-07-20T18:47:37.743 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:47:37.744 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:47:37.758 INFO:teuthology.orchestra.run.smithi192.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T18:47:37.760 INFO:teuthology.orchestra.run.smithi192.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T18:47:37.761 INFO:teuthology.orchestra.run.smithi192.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T18:47:37.762 INFO:teuthology.orchestra.run.smithi192.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T18:47:37.764 INFO:teuthology.orchestra.run.smithi192.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T18:47:37.767 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T18:47:37.768 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T18:47:37.900 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-07-20T18:47:37.913 INFO:teuthology.orchestra.run.smithi192.stdout:=========================================================================================================================== 2023-07-20T18:47:37.913 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-07-20T18:47:37.913 INFO:teuthology.orchestra.run.smithi192.stdout:=========================================================================================================================== 2023-07-20T18:47:37.913 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-07-20T18:47:37.913 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T18:47:37.913 INFO:teuthology.orchestra.run.smithi192.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T18:47:37.914 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T18:47:37.914 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T18:47:37.914 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T18:47:37.914 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T18:47:37.914 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T18:47:37.914 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T18:47:37.914 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T18:47:37.915 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T18:47:37.915 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T18:47:37.915 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T18:47:37.915 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T18:47:37.915 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T18:47:37.915 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T18:47:37.915 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T18:47:37.916 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T18:47:37.916 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T18:47:37.916 INFO:teuthology.orchestra.run.smithi192.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T18:47:37.916 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T18:47:37.916 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T18:47:37.916 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T18:47:37.916 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:47:37.917 INFO:teuthology.orchestra.run.smithi192.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T18:47:37.917 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T18:47:37.917 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T18:47:37.917 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T18:47:37.917 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T18:47:37.917 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T18:47:37.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T18:47:37.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T18:47:37.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T18:47:37.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T18:47:37.919 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T18:47:37.919 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T18:47:37.919 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T18:47:37.919 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T18:47:37.919 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2023-07-20T18:47:37.919 INFO:teuthology.orchestra.run.smithi192.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T18:47:37.919 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T18:47:37.919 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T18:47:37.920 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-07-20T18:47:37.920 INFO:teuthology.orchestra.run.smithi192.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T18:47:37.920 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T18:47:37.920 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T18:47:37.920 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T18:47:37.921 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T18:47:37.921 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T18:47:37.921 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T18:47:37.921 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T18:47:37.921 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T18:47:37.922 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T18:47:37.922 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T18:47:37.922 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T18:47:37.922 INFO:teuthology.orchestra.run.smithi192.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T18:47:37.922 INFO:teuthology.orchestra.run.smithi192.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T18:47:37.922 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:47:37.922 INFO:teuthology.orchestra.run.smithi192.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T18:47:37.923 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T18:47:37.923 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T18:47:37.923 INFO:teuthology.orchestra.run.smithi192.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T18:47:37.923 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T18:47:37.923 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T18:47:37.923 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T18:47:37.923 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T18:47:37.924 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T18:47:37.924 INFO:teuthology.orchestra.run.smithi192.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T18:47:37.924 INFO:teuthology.orchestra.run.smithi192.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T18:47:37.924 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T18:47:37.924 INFO:teuthology.orchestra.run.smithi192.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T18:47:37.924 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T18:47:37.924 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T18:47:37.925 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T18:47:37.925 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T18:47:37.925 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T18:47:37.925 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T18:47:37.925 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T18:47:37.925 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T18:47:37.925 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T18:47:37.926 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T18:47:37.926 INFO:teuthology.orchestra.run.smithi192.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T18:47:37.926 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T18:47:37.926 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T18:47:37.926 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T18:47:37.926 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T18:47:37.926 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T18:47:37.926 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T18:47:37.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T18:47:37.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T18:47:37.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T18:47:37.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T18:47:37.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T18:47:37.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T18:47:37.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T18:47:37.927 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T18:47:37.928 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:47:37.928 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T18:47:37.928 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T18:47:37.928 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T18:47:37.928 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T18:47:37.928 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T18:47:37.929 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T18:47:37.929 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T18:47:37.929 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T18:47:37.929 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T18:47:37.929 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T18:47:37.929 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T18:47:37.929 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T18:47:37.930 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T18:47:37.930 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T18:47:37.930 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T18:47:37.930 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T18:47:37.930 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T18:47:37.930 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T18:47:37.930 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:47:37.930 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T18:47:37.931 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T18:47:37.931 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T18:47:37.931 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T18:47:37.931 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T18:47:37.931 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T18:47:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T18:47:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T18:47:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T18:47:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T18:47:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T18:47:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T18:47:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T18:47:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T18:47:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T18:47:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T18:47:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T18:47:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T18:47:37.933 INFO:teuthology.orchestra.run.smithi192.stdout:Enabling module streams: 2023-07-20T18:47:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql 10 2023-07-20T18:47:37.934 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:47:37.934 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-07-20T18:47:37.934 INFO:teuthology.orchestra.run.smithi192.stdout:=========================================================================================================================== 2023-07-20T18:47:37.934 INFO:teuthology.orchestra.run.smithi192.stdout:Install 128 Packages 2023-07-20T18:47:37.934 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 3 Packages 2023-07-20T18:47:37.934 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:47:37.935 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 206 M 2023-07-20T18:47:37.935 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-07-20T18:47:38.388 INFO:teuthology.orchestra.run.smithi192.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 15 kB/s | 6.4 kB 00:00 2023-07-20T18:47:38.609 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:47:38.610 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:47:38.625 INFO:teuthology.orchestra.run.smithi088.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T18:47:38.626 INFO:teuthology.orchestra.run.smithi088.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T18:47:38.627 INFO:teuthology.orchestra.run.smithi088.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T18:47:38.628 INFO:teuthology.orchestra.run.smithi088.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T18:47:38.631 INFO:teuthology.orchestra.run.smithi088.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T18:47:38.634 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T18:47:38.635 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T18:47:38.768 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-20T18:47:38.781 INFO:teuthology.orchestra.run.smithi088.stdout:=========================================================================================================================== 2023-07-20T18:47:38.781 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-07-20T18:47:38.781 INFO:teuthology.orchestra.run.smithi088.stdout:=========================================================================================================================== 2023-07-20T18:47:38.781 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-07-20T18:47:38.781 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T18:47:38.781 INFO:teuthology.orchestra.run.smithi088.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T18:47:38.782 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T18:47:38.782 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T18:47:38.782 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T18:47:38.782 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T18:47:38.782 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T18:47:38.782 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T18:47:38.783 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T18:47:38.783 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T18:47:38.783 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T18:47:38.783 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T18:47:38.783 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T18:47:38.784 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T18:47:38.784 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T18:47:38.784 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T18:47:38.784 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T18:47:38.784 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T18:47:38.785 INFO:teuthology.orchestra.run.smithi088.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T18:47:38.786 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T18:47:38.786 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T18:47:38.786 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T18:47:38.786 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:47:38.786 INFO:teuthology.orchestra.run.smithi088.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T18:47:38.786 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T18:47:38.787 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T18:47:38.787 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T18:47:38.787 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T18:47:38.787 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T18:47:38.787 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T18:47:38.787 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T18:47:38.787 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T18:47:38.787 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T18:47:38.788 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T18:47:38.788 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T18:47:38.788 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T18:47:38.788 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T18:47:38.788 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2023-07-20T18:47:38.788 INFO:teuthology.orchestra.run.smithi088.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T18:47:38.788 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T18:47:38.789 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T18:47:38.789 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-07-20T18:47:38.789 INFO:teuthology.orchestra.run.smithi088.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T18:47:38.789 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T18:47:38.789 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T18:47:38.789 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T18:47:38.789 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T18:47:38.789 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T18:47:38.790 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T18:47:38.790 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T18:47:38.790 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T18:47:38.790 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T18:47:38.790 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T18:47:38.790 INFO:teuthology.orchestra.run.smithi088.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T18:47:38.790 INFO:teuthology.orchestra.run.smithi088.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T18:47:38.791 INFO:teuthology.orchestra.run.smithi088.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T18:47:38.791 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:47:38.791 INFO:teuthology.orchestra.run.smithi088.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T18:47:38.791 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T18:47:38.791 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T18:47:38.791 INFO:teuthology.orchestra.run.smithi088.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T18:47:38.791 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T18:47:38.792 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T18:47:38.792 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T18:47:38.792 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T18:47:38.792 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T18:47:38.793 INFO:teuthology.orchestra.run.smithi088.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T18:47:38.793 INFO:teuthology.orchestra.run.smithi088.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T18:47:38.793 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T18:47:38.793 INFO:teuthology.orchestra.run.smithi088.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T18:47:38.793 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T18:47:38.794 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T18:47:38.794 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T18:47:38.794 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T18:47:38.794 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T18:47:38.794 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T18:47:38.794 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T18:47:38.795 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T18:47:38.795 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T18:47:38.795 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T18:47:38.796 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T18:47:38.796 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T18:47:38.796 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T18:47:38.796 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T18:47:38.796 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T18:47:38.797 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T18:47:38.797 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T18:47:38.797 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T18:47:38.797 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T18:47:38.797 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T18:47:38.797 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T18:47:38.797 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T18:47:38.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T18:47:38.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T18:47:38.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T18:47:38.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:47:38.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T18:47:38.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T18:47:38.798 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T18:47:38.799 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T18:47:38.799 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T18:47:38.799 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T18:47:38.799 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T18:47:38.799 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T18:47:38.799 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T18:47:38.800 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T18:47:38.800 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T18:47:38.800 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T18:47:38.800 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T18:47:38.800 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T18:47:38.800 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T18:47:38.800 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T18:47:38.801 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T18:47:38.801 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T18:47:38.801 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:47:38.801 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T18:47:38.801 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T18:47:38.802 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T18:47:38.802 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T18:47:38.802 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T18:47:38.802 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T18:47:38.802 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T18:47:38.802 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T18:47:38.803 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T18:47:38.803 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T18:47:38.803 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T18:47:38.803 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T18:47:38.803 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T18:47:38.803 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T18:47:38.803 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T18:47:38.804 INFO:teuthology.orchestra.run.smithi088.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T18:47:38.804 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T18:47:38.804 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T18:47:38.804 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling module streams: 2023-07-20T18:47:38.804 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql 10 2023-07-20T18:47:38.804 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:47:38.804 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-07-20T18:47:38.805 INFO:teuthology.orchestra.run.smithi088.stdout:=========================================================================================================================== 2023-07-20T18:47:38.805 INFO:teuthology.orchestra.run.smithi088.stdout:Install 128 Packages 2023-07-20T18:47:38.805 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 3 Packages 2023-07-20T18:47:38.805 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:47:38.806 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 206 M 2023-07-20T18:47:38.806 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-07-20T18:47:39.113 INFO:teuthology.orchestra.run.smithi192.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.1 MB/s | 850 kB 00:00 2023-07-20T18:47:39.238 INFO:teuthology.orchestra.run.smithi192.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 1.3 MB/s | 167 kB 00:00 2023-07-20T18:47:39.800 INFO:teuthology.orchestra.run.smithi192.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 4.1 MB/s | 2.3 MB 00:00 2023-07-20T18:47:39.865 INFO:teuthology.orchestra.run.smithi192.stdout:(5/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 2.8 MB/s | 5.4 MB 00:01 2023-07-20T18:47:39.976 INFO:teuthology.orchestra.run.smithi088.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 8.5 kB/s | 6.4 kB 00:00 2023-07-20T18:47:40.199 INFO:teuthology.orchestra.run.smithi192.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 3.9 MB/s | 1.6 MB 00:00 2023-07-20T18:47:40.773 INFO:teuthology.orchestra.run.smithi192.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 5.1 MB/s | 4.6 MB 00:00 2023-07-20T18:47:41.094 INFO:teuthology.orchestra.run.smithi088.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 760 kB/s | 850 kB 00:01 2023-07-20T18:47:41.354 INFO:teuthology.orchestra.run.smithi088.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 647 kB/s | 167 kB 00:00 2023-07-20T18:47:41.486 INFO:teuthology.orchestra.run.smithi088.stdout:(4/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 2.4 MB/s | 5.4 MB 00:02 2023-07-20T18:47:41.994 INFO:teuthology.orchestra.run.smithi088.stdout:(5/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 3.1 MB/s | 1.6 MB 00:00 2023-07-20T18:47:42.645 INFO:teuthology.orchestra.run.smithi088.stdout:(6/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 1.8 MB/s | 2.3 MB 00:01 2023-07-20T18:47:42.648 INFO:teuthology.orchestra.run.smithi192.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 4.7 MB/s | 22 MB 00:04 2023-07-20T18:47:42.765 INFO:teuthology.orchestra.run.smithi192.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 208 kB/s | 24 kB 00:00 2023-07-20T18:47:42.951 INFO:teuthology.orchestra.run.smithi057.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:47:42.953 INFO:teuthology.orchestra.run.smithi057.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:47:42.968 INFO:teuthology.orchestra.run.smithi057.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T18:47:42.969 INFO:teuthology.orchestra.run.smithi057.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T18:47:42.970 INFO:teuthology.orchestra.run.smithi057.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T18:47:42.972 INFO:teuthology.orchestra.run.smithi057.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T18:47:42.974 INFO:teuthology.orchestra.run.smithi057.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T18:47:42.978 INFO:teuthology.orchestra.run.smithi057.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T18:47:42.979 INFO:teuthology.orchestra.run.smithi057.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T18:47:43.132 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-07-20T18:47:43.146 INFO:teuthology.orchestra.run.smithi057.stdout:=========================================================================================================================== 2023-07-20T18:47:43.146 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-07-20T18:47:43.146 INFO:teuthology.orchestra.run.smithi057.stdout:=========================================================================================================================== 2023-07-20T18:47:43.147 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-07-20T18:47:43.147 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T18:47:43.147 INFO:teuthology.orchestra.run.smithi057.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T18:47:43.147 INFO:teuthology.orchestra.run.smithi057.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T18:47:43.147 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T18:47:43.147 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T18:47:43.148 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T18:47:43.148 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T18:47:43.148 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T18:47:43.148 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T18:47:43.148 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T18:47:43.148 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T18:47:43.149 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T18:47:43.149 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T18:47:43.149 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T18:47:43.149 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T18:47:43.149 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T18:47:43.149 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T18:47:43.149 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T18:47:43.150 INFO:teuthology.orchestra.run.smithi057.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T18:47:43.150 INFO:teuthology.orchestra.run.smithi057.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T18:47:43.150 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T18:47:43.150 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T18:47:43.150 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:47:43.150 INFO:teuthology.orchestra.run.smithi057.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T18:47:43.151 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T18:47:43.151 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T18:47:43.151 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T18:47:43.151 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T18:47:43.151 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T18:47:43.151 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T18:47:43.152 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T18:47:43.152 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T18:47:43.152 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T18:47:43.152 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T18:47:43.152 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T18:47:43.152 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T18:47:43.153 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T18:47:43.153 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrading: 2023-07-20T18:47:43.153 INFO:teuthology.orchestra.run.smithi057.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T18:47:43.153 INFO:teuthology.orchestra.run.smithi057.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T18:47:43.153 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T18:47:43.153 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-07-20T18:47:43.154 INFO:teuthology.orchestra.run.smithi057.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T18:47:43.154 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T18:47:43.154 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T18:47:43.154 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T18:47:43.155 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T18:47:43.155 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T18:47:43.155 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T18:47:43.156 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T18:47:43.156 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T18:47:43.156 INFO:teuthology.orchestra.run.smithi057.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T18:47:43.156 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T18:47:43.156 INFO:teuthology.orchestra.run.smithi057.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T18:47:43.156 INFO:teuthology.orchestra.run.smithi057.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T18:47:43.156 INFO:teuthology.orchestra.run.smithi057.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T18:47:43.157 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:47:43.157 INFO:teuthology.orchestra.run.smithi057.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T18:47:43.157 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T18:47:43.157 INFO:teuthology.orchestra.run.smithi057.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T18:47:43.157 INFO:teuthology.orchestra.run.smithi057.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T18:47:43.158 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T18:47:43.158 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T18:47:43.158 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T18:47:43.158 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T18:47:43.158 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T18:47:43.158 INFO:teuthology.orchestra.run.smithi057.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T18:47:43.159 INFO:teuthology.orchestra.run.smithi057.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T18:47:43.159 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T18:47:43.159 INFO:teuthology.orchestra.run.smithi057.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T18:47:43.159 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T18:47:43.159 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T18:47:43.159 INFO:teuthology.orchestra.run.smithi057.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T18:47:43.160 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T18:47:43.160 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T18:47:43.160 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T18:47:43.160 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T18:47:43.160 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T18:47:43.160 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T18:47:43.161 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T18:47:43.161 INFO:teuthology.orchestra.run.smithi057.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T18:47:43.161 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T18:47:43.161 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T18:47:43.161 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T18:47:43.161 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T18:47:43.162 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T18:47:43.162 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T18:47:43.162 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T18:47:43.162 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T18:47:43.162 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T18:47:43.162 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T18:47:43.162 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T18:47:43.163 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T18:47:43.163 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T18:47:43.163 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T18:47:43.163 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:47:43.163 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T18:47:43.163 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T18:47:43.163 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T18:47:43.164 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T18:47:43.164 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T18:47:43.164 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T18:47:43.164 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T18:47:43.164 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T18:47:43.164 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T18:47:43.164 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T18:47:43.164 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T18:47:43.165 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T18:47:43.165 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T18:47:43.165 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T18:47:43.165 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T18:47:43.165 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T18:47:43.165 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T18:47:43.165 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T18:47:43.166 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:47:43.166 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T18:47:43.166 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T18:47:43.166 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T18:47:43.166 INFO:teuthology.orchestra.run.smithi057.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T18:47:43.166 INFO:teuthology.orchestra.run.smithi057.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T18:47:43.166 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T18:47:43.166 INFO:teuthology.orchestra.run.smithi057.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T18:47:43.167 INFO:teuthology.orchestra.run.smithi057.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T18:47:43.167 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T18:47:43.167 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T18:47:43.167 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T18:47:43.167 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T18:47:43.167 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T18:47:43.167 INFO:teuthology.orchestra.run.smithi057.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T18:47:43.167 INFO:teuthology.orchestra.run.smithi057.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T18:47:43.168 INFO:teuthology.orchestra.run.smithi057.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T18:47:43.168 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T18:47:43.168 INFO:teuthology.orchestra.run.smithi057.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T18:47:43.168 INFO:teuthology.orchestra.run.smithi057.stdout:Enabling module streams: 2023-07-20T18:47:43.168 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql 10 2023-07-20T18:47:43.168 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:47:43.168 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-07-20T18:47:43.169 INFO:teuthology.orchestra.run.smithi057.stdout:=========================================================================================================================== 2023-07-20T18:47:43.169 INFO:teuthology.orchestra.run.smithi057.stdout:Install 128 Packages 2023-07-20T18:47:43.169 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrade 3 Packages 2023-07-20T18:47:43.169 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:47:43.170 INFO:teuthology.orchestra.run.smithi088.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 3.9 MB/s | 4.6 MB 00:01 2023-07-20T18:47:43.170 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 206 M 2023-07-20T18:47:43.170 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-07-20T18:47:43.215 INFO:teuthology.orchestra.run.smithi192.stdout:(10/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 5.5 MB/s | 14 MB 00:02 2023-07-20T18:47:43.341 INFO:teuthology.orchestra.run.smithi192.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 2.0 MB/s | 260 kB 00:00 2023-07-20T18:47:43.533 INFO:teuthology.orchestra.run.smithi192.stdout:(12/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 5.3 MB/s | 18 MB 00:03 2023-07-20T18:47:43.558 INFO:teuthology.orchestra.run.smithi192.stdout:(13/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 143 kB/s | 31 kB 00:00 2023-07-20T18:47:43.692 INFO:teuthology.orchestra.run.smithi192.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.0 MB/s | 139 kB 00:00 2023-07-20T18:47:43.742 INFO:teuthology.orchestra.run.smithi192.stdout:(15/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 3.6 MB/s | 771 kB 00:00 2023-07-20T18:47:43.801 INFO:teuthology.orchestra.run.smithi192.stdout:(16/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.2 MB/s | 139 kB 00:00 2023-07-20T18:47:43.851 INFO:teuthology.orchestra.run.smithi192.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 4.2 MB/s | 472 kB 00:00 2023-07-20T18:47:43.951 INFO:teuthology.orchestra.run.smithi192.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 447 kB/s | 45 kB 00:00 2023-07-20T18:47:44.050 INFO:teuthology.orchestra.run.smithi057.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 18 kB/s | 6.4 kB 00:00 2023-07-20T18:47:44.076 INFO:teuthology.orchestra.run.smithi192.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 979 kB/s | 122 kB 00:00 2023-07-20T18:47:44.207 INFO:teuthology.orchestra.run.smithi192.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 1.5 MB/s | 197 kB 00:00 2023-07-20T18:47:44.310 INFO:teuthology.orchestra.run.smithi192.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 3.5 MB/s | 370 kB 00:00 2023-07-20T18:47:44.419 INFO:teuthology.orchestra.run.smithi192.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 3.3 MB/s | 363 kB 00:00 2023-07-20T18:47:44.502 INFO:teuthology.orchestra.run.smithi192.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.2 MB/s | 107 kB 00:00 2023-07-20T18:47:44.595 INFO:teuthology.orchestra.run.smithi192.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.0 MB/s | 94 kB 00:00 2023-07-20T18:47:45.011 INFO:teuthology.orchestra.run.smithi192.stdout:(25/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 5.6 MB/s | 6.8 MB 00:01 2023-07-20T18:47:45.033 INFO:teuthology.orchestra.run.smithi057.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 864 kB/s | 850 kB 00:00 2023-07-20T18:47:45.129 INFO:teuthology.orchestra.run.smithi192.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.5 MB/s | 179 kB 00:00 2023-07-20T18:47:45.209 INFO:teuthology.orchestra.run.smithi057.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 956 kB/s | 167 kB 00:00 2023-07-20T18:47:45.254 INFO:teuthology.orchestra.run.smithi192.stdout:(27/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 5.4 MB/s | 3.5 MB 00:00 2023-07-20T18:47:45.279 INFO:teuthology.orchestra.run.smithi192.stdout:(28/131): ceph-grafana-dashboards-18.0.0-5040.g 153 kB/s | 23 kB 00:00 2023-07-20T18:47:45.371 INFO:teuthology.orchestra.run.smithi192.stdout:(29/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.1 MB/s | 129 kB 00:00 2023-07-20T18:47:45.437 INFO:teuthology.orchestra.run.smithi088.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 3.6 MB/s | 22 MB 00:06 2023-07-20T18:47:45.545 INFO:teuthology.orchestra.run.smithi088.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 225 kB/s | 24 kB 00:00 2023-07-20T18:47:45.663 INFO:teuthology.orchestra.run.smithi192.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 4.5 MB/s | 1.7 MB 00:00 2023-07-20T18:47:45.788 INFO:teuthology.orchestra.run.smithi192.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 1.9 MB/s | 241 kB 00:00 2023-07-20T18:47:45.897 INFO:teuthology.orchestra.run.smithi192.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 437 kB/s | 47 kB 00:00 2023-07-20T18:47:46.014 INFO:teuthology.orchestra.run.smithi192.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 124 kB/s | 15 kB 00:00 2023-07-20T18:47:46.131 INFO:teuthology.orchestra.run.smithi192.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 2.2 MB/s | 258 kB 00:00 2023-07-20T18:47:46.240 INFO:teuthology.orchestra.run.smithi192.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.5 MB/s | 165 kB 00:00 2023-07-20T18:47:46.348 INFO:teuthology.orchestra.run.smithi192.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 166 kB/s | 18 kB 00:00 2023-07-20T18:47:46.433 INFO:teuthology.orchestra.run.smithi057.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 1.9 MB/s | 2.3 MB 00:01 2023-07-20T18:47:46.665 INFO:teuthology.orchestra.run.smithi192.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 5.7 MB/s | 7.4 MB 00:01 2023-07-20T18:47:46.867 INFO:teuthology.orchestra.run.smithi057.stdout:(5/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 1.7 MB/s | 5.4 MB 00:03 2023-07-20T18:47:47.034 INFO:teuthology.orchestra.run.smithi057.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 2.6 MB/s | 1.6 MB 00:00 2023-07-20T18:47:47.257 INFO:teuthology.orchestra.run.smithi192.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 337 kB/s | 306 kB 00:00 2023-07-20T18:47:47.283 INFO:teuthology.orchestra.run.smithi192.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 95 kB/s | 59 kB 00:00 2023-07-20T18:47:47.400 INFO:teuthology.orchestra.run.smithi192.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 527 kB/s | 75 kB 00:00 2023-07-20T18:47:47.534 INFO:teuthology.orchestra.run.smithi192.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 327 kB/s | 44 kB 00:00 2023-07-20T18:47:47.545 INFO:teuthology.orchestra.run.smithi088.stdout:(10/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 3.6 MB/s | 18 MB 00:04 2023-07-20T18:47:47.693 INFO:teuthology.orchestra.run.smithi192.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.1 MB/s | 185 kB 00:00 2023-07-20T18:47:47.779 INFO:teuthology.orchestra.run.smithi088.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 1.1 MB/s | 260 kB 00:00 2023-07-20T18:47:47.784 INFO:teuthology.orchestra.run.smithi192.stdout:(43/131): python3-asyncssh-2.7.0-2.el8.noarch.r 949 kB/s | 476 kB 00:00 2023-07-20T18:47:47.843 INFO:teuthology.orchestra.run.smithi192.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 219 kB/s | 33 kB 00:00 2023-07-20T18:47:47.887 INFO:teuthology.orchestra.run.smithi088.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 286 kB/s | 31 kB 00:00 2023-07-20T18:47:47.918 INFO:teuthology.orchestra.run.smithi192.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 122 kB/s | 16 kB 00:00 2023-07-20T18:47:47.985 INFO:teuthology.orchestra.run.smithi057.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 4.1 MB/s | 4.6 MB 00:01 2023-07-20T18:47:47.993 INFO:teuthology.orchestra.run.smithi192.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.1 MB/s | 173 kB 00:00 2023-07-20T18:47:48.060 INFO:teuthology.orchestra.run.smithi192.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.6 MB/s | 384 kB 00:00 2023-07-20T18:47:48.135 INFO:teuthology.orchestra.run.smithi192.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 573 kB/s | 81 kB 00:00 2023-07-20T18:47:48.196 INFO:teuthology.orchestra.run.smithi088.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 2.4 MB/s | 771 kB 00:00 2023-07-20T18:47:48.202 INFO:teuthology.orchestra.run.smithi192.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2023-07-20T18:47:48.286 INFO:teuthology.orchestra.run.smithi192.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 117 kB/s | 18 kB 00:00 2023-07-20T18:47:48.313 INFO:teuthology.orchestra.run.smithi088.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.2 MB/s | 139 kB 00:00 2023-07-20T18:47:48.422 INFO:teuthology.orchestra.run.smithi088.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.3 MB/s | 139 kB 00:00 2023-07-20T18:47:48.428 INFO:teuthology.orchestra.run.smithi192.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 344 kB/s | 49 kB 00:00 2023-07-20T18:47:48.461 INFO:teuthology.orchestra.run.smithi192.stdout:(52/131): python3-kubernetes-11.0.0-6.el8.noarc 4.5 MB/s | 1.2 MB 00:00 2023-07-20T18:47:48.538 INFO:teuthology.orchestra.run.smithi192.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 541 kB/s | 59 kB 00:00 2023-07-20T18:47:48.570 INFO:teuthology.orchestra.run.smithi192.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 554 kB/s | 60 kB 00:00 2023-07-20T18:47:48.597 INFO:teuthology.orchestra.run.smithi088.stdout:(16/131): libradosstriper1-18.0.0-5040.g1b0b2eb 2.6 MB/s | 472 kB 00:00 2023-07-20T18:47:48.671 INFO:teuthology.orchestra.run.smithi192.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2023-07-20T18:47:48.687 INFO:teuthology.orchestra.run.smithi192.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2023-07-20T18:47:48.821 INFO:teuthology.orchestra.run.smithi192.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 224 kB/s | 34 kB 00:00 2023-07-20T18:47:48.846 INFO:teuthology.orchestra.run.smithi192.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.2 MB/s | 196 kB 00:00 2023-07-20T18:47:48.938 INFO:teuthology.orchestra.run.smithi192.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 526 kB/s | 61 kB 00:00 2023-07-20T18:47:48.963 INFO:teuthology.orchestra.run.smithi192.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 160 kB/s | 19 kB 00:00 2023-07-20T18:47:49.064 INFO:teuthology.orchestra.run.smithi192.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 191 kB/s | 24 kB 00:00 2023-07-20T18:47:49.090 INFO:teuthology.orchestra.run.smithi192.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 233 kB/s | 29 kB 00:00 2023-07-20T18:47:49.190 INFO:teuthology.orchestra.run.smithi192.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 219 kB/s | 27 kB 00:00 2023-07-20T18:47:49.231 INFO:teuthology.orchestra.run.smithi192.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 1.8 MB/s | 255 kB 00:00 2023-07-20T18:47:49.323 INFO:teuthology.orchestra.run.smithi192.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.8 MB/s | 251 kB 00:00 2023-07-20T18:47:49.398 INFO:teuthology.orchestra.run.smithi192.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 367 kB/s | 61 kB 00:00 2023-07-20T18:47:49.449 INFO:teuthology.orchestra.run.smithi192.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 680 kB/s | 85 kB 00:00 2023-07-20T18:47:49.533 INFO:teuthology.orchestra.run.smithi192.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 171 kB/s | 23 kB 00:00 2023-07-20T18:47:49.558 INFO:teuthology.orchestra.run.smithi192.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T18:47:49.692 INFO:teuthology.orchestra.run.smithi192.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 351 kB/s | 47 kB 00:00 2023-07-20T18:47:49.809 INFO:teuthology.orchestra.run.smithi192.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.7 MB/s | 200 kB 00:00 2023-07-20T18:47:49.984 INFO:teuthology.orchestra.run.smithi192.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 243 kB/s | 43 kB 00:00 2023-07-20T18:47:50.018 INFO:teuthology.orchestra.run.smithi192.stdout:(73/131): thrift-0.13.0-2.el8.x86_64.rpm 3.1 MB/s | 1.7 MB 00:00 2023-07-20T18:47:50.077 INFO:teuthology.orchestra.run.smithi192.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.3 MB/s | 220 kB 00:00 2023-07-20T18:47:50.168 INFO:teuthology.orchestra.run.smithi192.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 320 kB/s | 48 kB 00:00 2023-07-20T18:47:50.227 INFO:teuthology.orchestra.run.smithi192.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 579 kB/s | 87 kB 00:00 2023-07-20T18:47:50.311 INFO:teuthology.orchestra.run.smithi192.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 300 kB/s | 43 kB 00:00 2023-07-20T18:47:50.353 INFO:teuthology.orchestra.run.smithi192.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 870 kB/s | 109 kB 00:00 2023-07-20T18:47:50.439 INFO:teuthology.orchestra.run.smithi088.stdout:(17/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 1.9 MB/s | 14 MB 00:07 2023-07-20T18:47:50.453 INFO:teuthology.orchestra.run.smithi192.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.6 MB/s | 229 kB 00:00 2023-07-20T18:47:50.522 INFO:teuthology.orchestra.run.smithi088.stdout:(18/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 3.5 MB/s | 6.8 MB 00:01 2023-07-20T18:47:50.548 INFO:teuthology.orchestra.run.smithi088.stdout:(19/131): python3-ceph-argparse-18.0.0-5040.g1b 413 kB/s | 45 kB 00:00 2023-07-20T18:47:50.562 INFO:teuthology.orchestra.run.smithi192.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 172 kB/s | 19 kB 00:00 2023-07-20T18:47:50.595 INFO:teuthology.orchestra.run.smithi192.stdout:(81/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 2.3 MB/s | 560 kB 00:00 2023-07-20T18:47:50.603 INFO:teuthology.orchestra.run.smithi057.stdout:(8/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x86 5.0 MB/s | 18 MB 00:03 2023-07-20T18:47:50.614 INFO:teuthology.orchestra.run.smithi088.stdout:(20/131): python3-ceph-common-18.0.0-5040.g1b0b 1.3 MB/s | 122 kB 00:00 2023-07-20T18:47:50.698 INFO:teuthology.orchestra.run.smithi088.stdout:(21/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 1.3 MB/s | 197 kB 00:00 2023-07-20T18:47:50.788 INFO:teuthology.orchestra.run.smithi192.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 687 kB/s | 155 kB 00:00 2023-07-20T18:47:50.790 INFO:teuthology.orchestra.run.smithi088.stdout:(22/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 2.1 MB/s | 370 kB 00:00 2023-07-20T18:47:50.818 INFO:teuthology.orchestra.run.smithi057.stdout:(9/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 3.1 MB/s | 22 MB 00:07 2023-07-20T18:47:50.843 INFO:teuthology.orchestra.run.smithi057.stdout:(10/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el 101 kB/s | 24 kB 00:00 2023-07-20T18:47:50.921 INFO:teuthology.orchestra.run.smithi192.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 1.7 MB/s | 559 kB 00:00 2023-07-20T18:47:50.946 INFO:teuthology.orchestra.run.smithi192.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 113 kB/s | 18 kB 00:00 2023-07-20T18:47:51.002 INFO:teuthology.orchestra.run.smithi057.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 1.6 MB/s | 260 kB 00:00 2023-07-20T18:47:51.047 INFO:teuthology.orchestra.run.smithi192.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.7 MB/s | 605 kB 00:00 2023-07-20T18:47:51.064 INFO:teuthology.orchestra.run.smithi192.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.0 MB/s | 238 kB 00:00 2023-07-20T18:47:51.074 INFO:teuthology.orchestra.run.smithi088.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 376 kB/s | 107 kB 00:00 2023-07-20T18:47:51.099 INFO:teuthology.orchestra.run.smithi088.stdout:(24/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 905 kB/s | 363 kB 00:00 2023-07-20T18:47:51.110 INFO:teuthology.orchestra.run.smithi057.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 286 kB/s | 31 kB 00:00 2023-07-20T18:47:51.157 INFO:teuthology.orchestra.run.smithi088.stdout:(25/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.1 MB/s | 94 kB 00:00 2023-07-20T18:47:51.266 INFO:teuthology.orchestra.run.smithi088.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.6 MB/s | 179 kB 00:00 2023-07-20T18:47:51.303 INFO:teuthology.orchestra.run.smithi057.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 3.9 MB/s | 771 kB 00:00 2023-07-20T18:47:51.333 INFO:teuthology.orchestra.run.smithi088.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 346 kB/s | 23 kB 00:00 2023-07-20T18:47:51.395 INFO:teuthology.orchestra.run.smithi057.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.5 MB/s | 139 kB 00:00 2023-07-20T18:47:51.417 INFO:teuthology.orchestra.run.smithi088.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.5 MB/s | 129 kB 00:00 2023-07-20T18:47:51.478 INFO:teuthology.orchestra.run.smithi057.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.6 MB/s | 139 kB 00:00 2023-07-20T18:47:51.522 INFO:teuthology.orchestra.run.smithi192.stdout:(87/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 5.5 MB/s | 48 MB 00:08 2023-07-20T18:47:51.549 INFO:teuthology.orchestra.run.smithi192.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 149 kB/s | 72 kB 00:00 2023-07-20T18:47:51.631 INFO:teuthology.orchestra.run.smithi192.stdout:(89/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 4.0 MB/s | 2.3 MB 00:00 2023-07-20T18:47:51.657 INFO:teuthology.orchestra.run.smithi192.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 551 kB/s | 60 kB 00:00 2023-07-20T18:47:51.662 INFO:teuthology.orchestra.run.smithi057.stdout:(16/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 3.7 MB/s | 14 MB 00:03 2023-07-20T18:47:51.695 INFO:teuthology.orchestra.run.smithi057.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 2.1 MB/s | 472 kB 00:00 2023-07-20T18:47:51.732 INFO:teuthology.orchestra.run.smithi192.stdout:(91/131): fuse-2.9.7-15.el8.x86_64.rpm 396 kB/s | 83 kB 00:00 2023-07-20T18:47:51.766 INFO:teuthology.orchestra.run.smithi192.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 531 kB/s | 58 kB 00:00 2023-07-20T18:47:51.779 INFO:teuthology.orchestra.run.smithi057.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 538 kB/s | 45 kB 00:00 2023-07-20T18:47:51.791 INFO:teuthology.orchestra.run.smithi192.stdout:(93/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 617 kB/s | 98 kB 00:00 2023-07-20T18:47:51.826 INFO:teuthology.orchestra.run.smithi088.stdout:(29/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 4.2 MB/s | 1.7 MB 00:00 2023-07-20T18:47:51.866 INFO:teuthology.orchestra.run.smithi192.stdout:(94/131): libcom_err-devel-1.45.6-4.el8.x86_64. 288 kB/s | 39 kB 00:00 2023-07-20T18:47:51.871 INFO:teuthology.orchestra.run.smithi057.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 1.3 MB/s | 122 kB 00:00 2023-07-20T18:47:51.925 INFO:teuthology.orchestra.run.smithi192.stdout:(95/131): NetworkManager-initscripts-updown-1.3 2.3 MB/s | 138 kB 00:00 2023-07-20T18:47:51.950 INFO:teuthology.orchestra.run.smithi192.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.1 MB/s | 171 kB 00:00 2023-07-20T18:47:51.984 INFO:teuthology.orchestra.run.smithi192.stdout:(97/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 2.9 MB/s | 644 kB 00:00 2023-07-20T18:47:52.026 INFO:teuthology.orchestra.run.smithi192.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2023-07-20T18:47:52.051 INFO:teuthology.orchestra.run.smithi192.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 811 kB/s | 54 kB 00:00 2023-07-20T18:47:52.151 INFO:teuthology.orchestra.run.smithi192.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.0 MB/s | 259 kB 00:00 2023-07-20T18:47:52.185 INFO:teuthology.orchestra.run.smithi192.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2023-07-20T18:47:52.310 INFO:teuthology.orchestra.run.smithi192.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 5.4 MB/s | 688 kB 00:00 2023-07-20T18:47:52.322 INFO:teuthology.orchestra.run.smithi057.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 438 kB/s | 197 kB 00:00 2023-07-20T18:47:52.344 INFO:teuthology.orchestra.run.smithi192.stdout:(103/131): python3-markupsafe-0.23-19.el8.x86_6 202 kB/s | 39 kB 00:00 2023-07-20T18:47:52.446 INFO:teuthology.orchestra.run.smithi057.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 2.9 MB/s | 370 kB 00:00 2023-07-20T18:47:52.451 INFO:teuthology.orchestra.run.smithi088.stdout:(30/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 2.6 MB/s | 3.5 MB 00:01 2023-07-20T18:47:52.544 INFO:teuthology.orchestra.run.smithi192.stdout:(104/131): python3-scipy-1.0.0-21.module+el8.5. 22 MB/s | 14 MB 00:00 2023-07-20T18:47:52.569 INFO:teuthology.orchestra.run.smithi192.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 697 kB/s | 157 kB 00:00 2023-07-20T18:47:52.589 INFO:teuthology.orchestra.run.smithi057.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 2.5 MB/s | 363 kB 00:00 2023-07-20T18:47:52.593 INFO:teuthology.orchestra.run.smithi088.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 1.7 MB/s | 241 kB 00:00 2023-07-20T18:47:52.603 INFO:teuthology.orchestra.run.smithi192.stdout:(106/131): python3-jinja2-2.10.1-3.el8.noarch.r 1.8 MB/s | 538 kB 00:00 2023-07-20T18:47:52.681 INFO:teuthology.orchestra.run.smithi057.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.1 MB/s | 107 kB 00:00 2023-07-20T18:47:52.693 INFO:teuthology.orchestra.run.smithi088.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 475 kB/s | 47 kB 00:00 2023-07-20T18:47:52.703 INFO:teuthology.orchestra.run.smithi192.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-07-20T18:47:52.737 INFO:teuthology.orchestra.run.smithi192.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 4.2 MB/s | 713 kB 00:00 2023-07-20T18:47:52.764 INFO:teuthology.orchestra.run.smithi057.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.1 MB/s | 94 kB 00:00 2023-07-20T18:47:52.785 INFO:teuthology.orchestra.run.smithi088.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 159 kB/s | 15 kB 00:00 2023-07-20T18:47:52.887 INFO:teuthology.orchestra.run.smithi192.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2023-07-20T18:47:52.902 INFO:teuthology.orchestra.run.smithi088.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 2.2 MB/s | 258 kB 00:00 2023-07-20T18:47:52.904 INFO:teuthology.orchestra.run.smithi192.stdout:(110/131): python3-jsonpatch-1.21-2.el8.noarch. 132 kB/s | 27 kB 00:00 2023-07-20T18:47:52.938 INFO:teuthology.orchestra.run.smithi192.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 1.6 MB/s | 320 kB 00:00 2023-07-20T18:47:52.985 INFO:teuthology.orchestra.run.smithi088.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.9 MB/s | 165 kB 00:00 2023-07-20T18:47:53.015 INFO:teuthology.orchestra.run.smithi192.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 350 kB/s | 44 kB 00:00 2023-07-20T18:47:53.038 INFO:teuthology.orchestra.run.smithi192.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 4.5 MB/s | 457 kB 00:00 2023-07-20T18:47:53.061 INFO:teuthology.orchestra.run.smithi088.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 240 kB/s | 18 kB 00:00 2023-07-20T18:47:53.130 INFO:teuthology.orchestra.run.smithi192.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 17 MB/s | 3.7 MB 00:00 2023-07-20T18:47:53.155 INFO:teuthology.orchestra.run.smithi192.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 547 kB/s | 64 kB 00:00 2023-07-20T18:47:53.181 INFO:teuthology.orchestra.run.smithi192.stdout:(116/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.3 MB/s | 225 kB 00:00 2023-07-20T18:47:53.239 INFO:teuthology.orchestra.run.smithi192.stdout:(117/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.1 MB/s | 126 kB 00:00 2023-07-20T18:47:53.265 INFO:teuthology.orchestra.run.smithi192.stdout:(118/131): python3-jsonpointer-1.10-11.el8.noar 184 kB/s | 20 kB 00:00 2023-07-20T18:47:53.290 INFO:teuthology.orchestra.run.smithi057.stdout:(25/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 4.2 MB/s | 6.8 MB 00:01 2023-07-20T18:47:53.307 INFO:teuthology.orchestra.run.smithi192.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 825 kB/s | 103 kB 00:00 2023-07-20T18:47:53.381 INFO:teuthology.orchestra.run.smithi057.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.9 MB/s | 179 kB 00:00 2023-07-20T18:47:53.440 INFO:teuthology.orchestra.run.smithi192.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2023-07-20T18:47:53.448 INFO:teuthology.orchestra.run.smithi057.stdout:(27/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 5.2 MB/s | 3.5 MB 00:00 2023-07-20T18:47:53.457 INFO:teuthology.orchestra.run.smithi192.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 1.3 MB/s | 198 kB 00:00 2023-07-20T18:47:53.465 INFO:teuthology.orchestra.run.smithi057.stdout:(28/131): ceph-grafana-dashboards-18.0.0-5040.g 276 kB/s | 23 kB 00:00 2023-07-20T18:47:53.532 INFO:teuthology.orchestra.run.smithi057.stdout:(29/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.5 MB/s | 129 kB 00:00 2023-07-20T18:47:53.566 INFO:teuthology.orchestra.run.smithi192.stdout:(122/131): openblas-threads-0.3.15-3.el8.x86_64 16 MB/s | 4.9 MB 00:00 2023-07-20T18:47:53.674 INFO:teuthology.orchestra.run.smithi192.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2023-07-20T18:47:53.686 INFO:teuthology.orchestra.run.smithi088.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 4.0 MB/s | 7.4 MB 00:01 2023-07-20T18:47:53.699 INFO:teuthology.orchestra.run.smithi192.stdout:(124/131): jq-1.6-3.el8.x86_64.rpm 834 kB/s | 202 kB 00:00 2023-07-20T18:47:53.725 INFO:teuthology.orchestra.run.smithi192.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.2 MB/s | 354 kB 00:00 2023-07-20T18:47:53.809 INFO:teuthology.orchestra.run.smithi192.stdout:(126/131): postgresql-contrib-10.21-2.module+el 7.2 MB/s | 808 kB 00:00 2023-07-20T18:47:53.820 INFO:teuthology.orchestra.run.smithi088.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 404 kB/s | 306 kB 00:00 2023-07-20T18:47:53.866 INFO:teuthology.orchestra.run.smithi057.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 4.3 MB/s | 1.7 MB 00:00 2023-07-20T18:47:53.892 INFO:teuthology.orchestra.run.smithi192.stdout:(127/131): postgresql-server-10.21-2.module+el8 23 MB/s | 5.1 MB 00:00 2023-07-20T18:47:53.943 INFO:teuthology.orchestra.run.smithi192.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 6.9 MB/s | 1.5 MB 00:00 2023-07-20T18:47:53.944 INFO:teuthology.orchestra.run.smithi088.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 601 kB/s | 75 kB 00:00 2023-07-20T18:47:53.966 INFO:teuthology.orchestra.run.smithi057.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 2.4 MB/s | 241 kB 00:00 2023-07-20T18:47:54.033 INFO:teuthology.orchestra.run.smithi057.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 711 kB/s | 47 kB 00:00 2023-07-20T18:47:54.101 INFO:teuthology.orchestra.run.smithi192.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.9 MB/s | 798 kB 00:00 2023-07-20T18:47:54.117 INFO:teuthology.orchestra.run.smithi057.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 175 kB/s | 15 kB 00:00 2023-07-20T18:47:54.187 INFO:teuthology.orchestra.run.smithi088.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.9 MB/s | 476 kB 00:00 2023-07-20T18:47:54.234 INFO:teuthology.orchestra.run.smithi057.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 2.2 MB/s | 258 kB 00:00 2023-07-20T18:47:54.237 INFO:teuthology.orchestra.run.smithi088.stdout:(41/131): liboath-2.6.2-3.el8.x86_64.rpm 107 kB/s | 59 kB 00:00 2023-07-20T18:47:54.430 INFO:teuthology.orchestra.run.smithi088.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 180 kB/s | 44 kB 00:00 2023-07-20T18:47:54.472 INFO:teuthology.orchestra.run.smithi088.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 793 kB/s | 185 kB 00:00 2023-07-20T18:47:54.572 INFO:teuthology.orchestra.run.smithi088.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 233 kB/s | 33 kB 00:00 2023-07-20T18:47:54.614 INFO:teuthology.orchestra.run.smithi088.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 115 kB/s | 16 kB 00:00 2023-07-20T18:47:54.643 INFO:teuthology.orchestra.run.smithi057.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 404 kB/s | 165 kB 00:00 2023-07-20T18:47:54.661 INFO:teuthology.orchestra.run.smithi192.stdout:(130/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 4.3 MB/s | 3.7 MB 00:00 2023-07-20T18:47:54.706 INFO:teuthology.orchestra.run.smithi088.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2023-07-20T18:47:54.735 INFO:teuthology.orchestra.run.smithi057.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 197 kB/s | 18 kB 00:00 2023-07-20T18:47:54.760 INFO:teuthology.orchestra.run.smithi192.stdout:(131/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 4.2 MB/s | 3.6 MB 00:00 2023-07-20T18:47:54.764 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:47:54.764 INFO:teuthology.orchestra.run.smithi192.stdout:Total 12 MB/s | 206 MB 00:16 2023-07-20T18:47:54.856 INFO:teuthology.orchestra.run.smithi088.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 542 kB/s | 81 kB 00:00 2023-07-20T18:47:54.990 INFO:teuthology.orchestra.run.smithi088.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2023-07-20T18:47:55.098 INFO:teuthology.orchestra.run.smithi088.stdout:(49/131): python3-cherrypy-18.4.0-1.el8.noarch. 795 kB/s | 384 kB 00:00 2023-07-20T18:47:55.132 INFO:teuthology.orchestra.run.smithi088.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 124 kB/s | 18 kB 00:00 2023-07-20T18:47:55.168 INFO:teuthology.orchestra.run.smithi057.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 4.5 MB/s | 7.4 MB 00:01 2023-07-20T18:47:55.266 INFO:teuthology.orchestra.run.smithi088.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 365 kB/s | 49 kB 00:00 2023-07-20T18:47:55.391 INFO:teuthology.orchestra.run.smithi088.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 470 kB/s | 59 kB 00:00 2023-07-20T18:47:55.460 INFO:teuthology.orchestra.run.smithi057.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 422 kB/s | 306 kB 00:00 2023-07-20T18:47:55.532 INFO:teuthology.orchestra.run.smithi088.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 426 kB/s | 60 kB 00:00 2023-07-20T18:47:55.581 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-07-20T18:47:55.644 INFO:teuthology.orchestra.run.smithi057.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 409 kB/s | 75 kB 00:00 2023-07-20T18:47:55.683 INFO:teuthology.orchestra.run.smithi088.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2023-07-20T18:47:55.701 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-07-20T18:47:55.701 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-07-20T18:47:55.825 INFO:teuthology.orchestra.run.smithi088.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 110 kB/s | 16 kB 00:00 2023-07-20T18:47:55.886 INFO:teuthology.orchestra.run.smithi057.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 82 kB/s | 59 kB 00:00 2023-07-20T18:47:55.967 INFO:teuthology.orchestra.run.smithi088.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 237 kB/s | 34 kB 00:00 2023-07-20T18:47:56.053 INFO:teuthology.orchestra.run.smithi057.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 262 kB/s | 44 kB 00:00 2023-07-20T18:47:56.194 INFO:teuthology.orchestra.run.smithi057.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 865 kB/s | 476 kB 00:00 2023-07-20T18:47:56.220 INFO:teuthology.orchestra.run.smithi088.stdout:(57/131): python3-kubernetes-11.0.0-6.el8.noarc 1.0 MB/s | 1.2 MB 00:01 2023-07-20T18:47:56.250 INFO:teuthology.orchestra.run.smithi088.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 693 kB/s | 196 kB 00:00 2023-07-20T18:47:56.278 INFO:teuthology.orchestra.run.smithi057.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 823 kB/s | 185 kB 00:00 2023-07-20T18:47:56.303 INFO:teuthology.orchestra.run.smithi057.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 304 kB/s | 33 kB 00:00 2023-07-20T18:47:56.342 INFO:teuthology.orchestra.run.smithi088.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 505 kB/s | 61 kB 00:00 2023-07-20T18:47:56.362 INFO:teuthology.orchestra.run.smithi057.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 195 kB/s | 16 kB 00:00 2023-07-20T18:47:56.368 INFO:teuthology.orchestra.run.smithi088.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 160 kB/s | 19 kB 00:00 2023-07-20T18:47:56.476 INFO:teuthology.orchestra.run.smithi088.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 179 kB/s | 24 kB 00:00 2023-07-20T18:47:56.502 INFO:teuthology.orchestra.run.smithi088.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 219 kB/s | 29 kB 00:00 2023-07-20T18:47:56.537 INFO:teuthology.orchestra.run.smithi057.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 740 kB/s | 173 kB 00:00 2023-07-20T18:47:56.619 INFO:teuthology.orchestra.run.smithi088.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 194 kB/s | 27 kB 00:00 2023-07-20T18:47:56.661 INFO:teuthology.orchestra.run.smithi088.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 1.6 MB/s | 255 kB 00:00 2023-07-20T18:47:56.729 INFO:teuthology.orchestra.run.smithi057.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 424 kB/s | 81 kB 00:00 2023-07-20T18:47:56.819 INFO:teuthology.orchestra.run.smithi088.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 387 kB/s | 61 kB 00:00 2023-07-20T18:47:56.821 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-07-20T18:47:56.825 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-07-20T18:47:56.829 INFO:teuthology.orchestra.run.smithi057.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-07-20T18:47:56.863 INFO:teuthology.orchestra.run.smithi057.stdout:(49/131): python3-cherrypy-18.4.0-1.el8.noarch. 768 kB/s | 384 kB 00:00 2023-07-20T18:47:56.878 INFO:teuthology.orchestra.run.smithi088.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 970 kB/s | 251 kB 00:00 2023-07-20T18:47:56.930 INFO:teuthology.orchestra.run.smithi057.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 175 kB/s | 18 kB 00:00 2023-07-20T18:47:56.978 INFO:teuthology.orchestra.run.smithi088.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 538 kB/s | 85 kB 00:00 2023-07-20T18:47:57.020 INFO:teuthology.orchestra.run.smithi088.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 161 kB/s | 23 kB 00:00 2023-07-20T18:47:57.039 INFO:teuthology.orchestra.run.smithi057.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 450 kB/s | 49 kB 00:00 2023-07-20T18:47:57.046 INFO:teuthology.orchestra.run.smithi088.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T18:47:57.139 INFO:teuthology.orchestra.run.smithi057.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 589 kB/s | 59 kB 00:00 2023-07-20T18:47:57.146 INFO:teuthology.orchestra.run.smithi088.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 468 kB/s | 47 kB 00:00 2023-07-20T18:47:57.256 INFO:teuthology.orchestra.run.smithi057.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 517 kB/s | 60 kB 00:00 2023-07-20T18:47:57.296 INFO:teuthology.orchestra.run.smithi088.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.3 MB/s | 200 kB 00:00 2023-07-20T18:47:57.430 INFO:teuthology.orchestra.run.smithi088.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 320 kB/s | 43 kB 00:00 2023-07-20T18:47:57.597 INFO:teuthology.orchestra.run.smithi088.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2023-07-20T18:47:57.607 INFO:teuthology.orchestra.run.smithi057.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 808 kB/s | 283 kB 00:00 2023-07-20T18:47:57.639 INFO:teuthology.orchestra.run.smithi088.stdout:(74/131): thrift-0.13.0-2.el8.x86_64.rpm 2.6 MB/s | 1.7 MB 00:00 2023-07-20T18:47:57.664 INFO:teuthology.orchestra.run.smithi088.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 718 kB/s | 48 kB 00:00 2023-07-20T18:47:57.707 INFO:teuthology.orchestra.run.smithi057.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2023-07-20T18:47:57.791 INFO:teuthology.orchestra.run.smithi057.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 405 kB/s | 34 kB 00:00 2023-07-20T18:47:57.806 INFO:teuthology.orchestra.run.smithi088.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 521 kB/s | 87 kB 00:00 2023-07-20T18:47:57.856 INFO:teuthology.orchestra.run.smithi088.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 229 kB/s | 43 kB 00:00 2023-07-20T18:47:57.957 INFO:teuthology.orchestra.run.smithi088.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 728 kB/s | 109 kB 00:00 2023-07-20T18:47:57.982 INFO:teuthology.orchestra.run.smithi088.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2023-07-20T18:47:57.999 INFO:teuthology.orchestra.run.smithi057.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 944 kB/s | 196 kB 00:00 2023-07-20T18:47:58.074 INFO:teuthology.orchestra.run.smithi057.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 824 kB/s | 61 kB 00:00 2023-07-20T18:47:58.099 INFO:teuthology.orchestra.run.smithi088.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 160 kB/s | 19 kB 00:00 2023-07-20T18:47:58.132 INFO:teuthology.orchestra.run.smithi088.stdout:(81/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.1 MB/s | 560 kB 00:00 2023-07-20T18:47:58.133 INFO:teuthology.orchestra.run.smithi057.stdout:(59/131): python3-kubernetes-11.0.0-6.el8.noarc 940 kB/s | 1.2 MB 00:01 2023-07-20T18:47:58.158 INFO:teuthology.orchestra.run.smithi057.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 224 kB/s | 19 kB 00:00 2023-07-20T18:47:58.209 INFO:teuthology.orchestra.run.smithi057.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 319 kB/s | 24 kB 00:00 2023-07-20T18:47:58.233 INFO:teuthology.orchestra.run.smithi088.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.1 MB/s | 155 kB 00:00 2023-07-20T18:47:58.234 INFO:teuthology.orchestra.run.smithi057.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 389 kB/s | 29 kB 00:00 2023-07-20T18:47:58.301 INFO:teuthology.orchestra.run.smithi057.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 299 kB/s | 27 kB 00:00 2023-07-20T18:47:58.342 INFO:teuthology.orchestra.run.smithi088.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 2.6 MB/s | 559 kB 00:00 2023-07-20T18:47:58.367 INFO:teuthology.orchestra.run.smithi088.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 134 kB/s | 18 kB 00:00 2023-07-20T18:47:58.468 INFO:teuthology.orchestra.run.smithi057.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 1.1 MB/s | 255 kB 00:00 2023-07-20T18:47:58.560 INFO:teuthology.orchestra.run.smithi057.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 663 kB/s | 61 kB 00:00 2023-07-20T18:47:58.652 INFO:teuthology.orchestra.run.smithi057.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 717 kB/s | 251 kB 00:00 2023-07-20T18:47:58.677 INFO:teuthology.orchestra.run.smithi057.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 732 kB/s | 85 kB 00:00 2023-07-20T18:47:58.727 INFO:teuthology.orchestra.run.smithi057.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 306 kB/s | 23 kB 00:00 2023-07-20T18:47:58.752 INFO:teuthology.orchestra.run.smithi057.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T18:47:58.809 INFO:teuthology.orchestra.run.smithi088.stdout:(85/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 3.6 MB/s | 48 MB 00:13 2023-07-20T18:47:58.842 INFO:teuthology.orchestra.run.smithi088.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 500 kB/s | 238 kB 00:00 2023-07-20T18:47:58.876 INFO:teuthology.orchestra.run.smithi088.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 1.1 MB/s | 605 kB 00:00 2023-07-20T18:47:58.894 INFO:teuthology.orchestra.run.smithi057.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 331 kB/s | 47 kB 00:00 2023-07-20T18:47:58.951 INFO:teuthology.orchestra.run.smithi088.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-07-20T18:47:59.035 INFO:teuthology.orchestra.run.smithi088.stdout:(89/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 387 kB/s | 72 kB 00:00 2023-07-20T18:47:59.045 INFO:teuthology.orchestra.run.smithi057.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.3 MB/s | 200 kB 00:00 2023-07-20T18:47:59.060 INFO:teuthology.orchestra.run.smithi088.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 551 kB/s | 60 kB 00:00 2023-07-20T18:47:59.160 INFO:teuthology.orchestra.run.smithi088.stdout:(91/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 6.6 MB/s | 2.3 MB 00:00 2023-07-20T18:47:59.170 INFO:teuthology.orchestra.run.smithi057.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 341 kB/s | 43 kB 00:00 2023-07-20T18:47:59.185 INFO:teuthology.orchestra.run.smithi088.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 460 kB/s | 58 kB 00:00 2023-07-20T18:47:59.211 INFO:teuthology.orchestra.run.smithi088.stdout:(93/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 558 kB/s | 98 kB 00:00 2023-07-20T18:47:59.303 INFO:teuthology.orchestra.run.smithi088.stdout:(94/131): libcom_err-devel-1.45.6-4.el8.x86_64. 272 kB/s | 39 kB 00:00 2023-07-20T18:47:59.303 INFO:teuthology.orchestra.run.smithi057.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2023-07-20T18:47:59.345 INFO:teuthology.orchestra.run.smithi088.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 4.0 MB/s | 644 kB 00:00 2023-07-20T18:47:59.371 INFO:teuthology.orchestra.run.smithi088.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.1 MB/s | 171 kB 00:00 2023-07-20T18:47:59.439 INFO:teuthology.orchestra.run.smithi057.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 356 kB/s | 48 kB 00:00 2023-07-20T18:47:59.471 INFO:teuthology.orchestra.run.smithi088.stdout:(97/131): NetworkManager-initscripts-updown-1.3 821 kB/s | 138 kB 00:00 2023-07-20T18:47:59.546 INFO:teuthology.orchestra.run.smithi088.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 628 kB/s | 110 kB 00:00 2023-07-20T18:47:59.596 INFO:teuthology.orchestra.run.smithi057.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 556 kB/s | 87 kB 00:00 2023-07-20T18:47:59.597 INFO:teuthology.orchestra.run.smithi088.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 433 kB/s | 54 kB 00:00 2023-07-20T18:47:59.646 INFO:teuthology.orchestra.run.smithi088.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2023-07-20T18:47:59.680 INFO:teuthology.orchestra.run.smithi088.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2023-07-20T18:47:59.755 INFO:teuthology.orchestra.run.smithi057.stdout:(76/131): thrift-0.13.0-2.el8.x86_64.rpm 1.6 MB/s | 1.7 MB 00:01 2023-07-20T18:47:59.764 INFO:teuthology.orchestra.run.smithi088.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 332 kB/s | 39 kB 00:00 2023-07-20T18:47:59.780 INFO:teuthology.orchestra.run.smithi057.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 233 kB/s | 43 kB 00:00 2023-07-20T18:47:59.922 INFO:teuthology.orchestra.run.smithi088.stdout:(103/131): python3-scipy-1.0.0-21.module+el8.5. 24 MB/s | 14 MB 00:00 2023-07-20T18:47:59.956 INFO:teuthology.orchestra.run.smithi088.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 2.7 MB/s | 538 kB 00:00 2023-07-20T18:47:59.981 INFO:teuthology.orchestra.run.smithi088.stdout:(105/131): bison-3.0.4-10.el8.x86_64.rpm 2.2 MB/s | 688 kB 00:00 2023-07-20T18:47:59.993 INFO:teuthology.orchestra.run.smithi057.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 459 kB/s | 109 kB 00:00 2023-07-20T18:48:00.048 INFO:teuthology.orchestra.run.smithi088.stdout:(106/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-07-20T18:48:00.082 INFO:teuthology.orchestra.run.smithi088.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2023-07-20T18:48:00.115 INFO:teuthology.orchestra.run.smithi088.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2023-07-20T18:48:00.172 INFO:teuthology.orchestra.run.smithi057.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 597 kB/s | 229 kB 00:00 2023-07-20T18:48:00.214 INFO:teuthology.orchestra.run.smithi057.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 2.5 MB/s | 560 kB 00:00 2023-07-20T18:48:00.215 INFO:teuthology.orchestra.run.smithi088.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 199 kB/s | 27 kB 00:00 2023-07-20T18:48:00.323 INFO:teuthology.orchestra.run.smithi057.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 124 kB/s | 19 kB 00:00 2023-07-20T18:48:00.324 INFO:teuthology.orchestra.run.smithi088.stdout:(110/131): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2023-07-20T18:48:00.348 INFO:teuthology.orchestra.run.smithi057.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.1 MB/s | 155 kB 00:00 2023-07-20T18:48:00.349 INFO:teuthology.orchestra.run.smithi088.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 329 kB/s | 44 kB 00:00 2023-07-20T18:48:00.375 INFO:teuthology.orchestra.run.smithi088.stdout:(112/131): flex-2.6.1-9.el8.x86_64.rpm 1.2 MB/s | 320 kB 00:00 2023-07-20T18:48:00.490 INFO:teuthology.orchestra.run.smithi057.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 126 kB/s | 18 kB 00:00 2023-07-20T18:48:00.500 INFO:teuthology.orchestra.run.smithi088.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.8 MB/s | 225 kB 00:00 2023-07-20T18:48:00.541 INFO:teuthology.orchestra.run.smithi057.stdout:(84/131): python3-cryptography-3.2.1-5.el8.x86_ 2.5 MB/s | 559 kB 00:00 2023-07-20T18:48:00.542 INFO:teuthology.orchestra.run.smithi088.stdout:(114/131): python3-werkzeug-0.12.2-4.el8.noarch 2.3 MB/s | 457 kB 00:00 2023-07-20T18:48:00.625 INFO:teuthology.orchestra.run.smithi088.stdout:(115/131): python3-numpy-1.14.3-10.el8.x86_64.r 12 MB/s | 3.7 MB 00:00 2023-07-20T18:48:00.642 INFO:teuthology.orchestra.run.smithi088.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 451 kB/s | 64 kB 00:00 2023-07-20T18:48:00.649 INFO:teuthology.orchestra.run.smithi057.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.7 MB/s | 605 kB 00:00 2023-07-20T18:48:00.674 INFO:teuthology.orchestra.run.smithi057.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.7 MB/s | 238 kB 00:00 2023-07-20T18:48:00.760 INFO:teuthology.orchestra.run.smithi088.stdout:(117/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 582 kB/s | 126 kB 00:00 2023-07-20T18:48:00.784 INFO:teuthology.orchestra.run.smithi088.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 729 kB/s | 103 kB 00:00 2023-07-20T18:48:00.792 INFO:teuthology.orchestra.run.smithi057.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 619 kB/s | 72 kB 00:00 2023-07-20T18:48:00.826 INFO:teuthology.orchestra.run.smithi088.stdout:(119/131): python3-jsonpointer-1.10-11.el8.noar 100 kB/s | 20 kB 00:00 2023-07-20T18:48:00.902 INFO:teuthology.orchestra.run.smithi088.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 4.9 MB/s | 709 kB 00:00 2023-07-20T18:48:00.917 INFO:teuthology.orchestra.run.smithi057.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 8.7 MB/s | 2.3 MB 00:00 2023-07-20T18:48:00.942 INFO:teuthology.orchestra.run.smithi057.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 552 kB/s | 83 kB 00:00 2023-07-20T18:48:01.019 INFO:teuthology.orchestra.run.smithi088.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 21 MB/s | 4.9 MB 00:00 2023-07-20T18:48:01.026 INFO:teuthology.orchestra.run.smithi057.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 551 kB/s | 60 kB 00:00 2023-07-20T18:48:01.044 INFO:teuthology.orchestra.run.smithi088.stdout:(122/131): libpq-13.5-1.el8.x86_64.rpm 911 kB/s | 198 kB 00:00 2023-07-20T18:48:01.086 INFO:teuthology.orchestra.run.smithi088.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 3.0 MB/s | 202 kB 00:00 2023-07-20T18:48:01.101 INFO:teuthology.orchestra.run.smithi057.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 619 kB/s | 98 kB 00:00 2023-07-20T18:48:01.135 INFO:teuthology.orchestra.run.smithi057.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 531 kB/s | 58 kB 00:00 2023-07-20T18:48:01.178 INFO:teuthology.orchestra.run.smithi088.stdout:(124/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-07-20T18:48:01.185 INFO:teuthology.orchestra.run.smithi057.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 461 kB/s | 39 kB 00:00 2023-07-20T18:48:01.220 INFO:teuthology.orchestra.run.smithi088.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.0 MB/s | 354 kB 00:00 2023-07-20T18:48:01.227 INFO:teuthology.orchestra.run.smithi057.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 6.9 MB/s | 644 kB 00:00 2023-07-20T18:48:01.279 INFO:teuthology.orchestra.run.smithi088.stdout:(126/131): postgresql-contrib-10.21-2.module+el 7.9 MB/s | 808 kB 00:00 2023-07-20T18:48:01.362 INFO:teuthology.orchestra.run.smithi088.stdout:(127/131): postgresql-server-10.21-2.module+el8 18 MB/s | 5.1 MB 00:00 2023-07-20T18:48:01.369 INFO:teuthology.orchestra.run.smithi057.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 934 kB/s | 171 kB 00:00 2023-07-20T18:48:01.421 INFO:teuthology.orchestra.run.smithi088.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 7.5 MB/s | 1.5 MB 00:00 2023-07-20T18:48:01.428 INFO:teuthology.orchestra.run.smithi057.stdout:(96/131): NetworkManager-initscripts-updown-1.3 687 kB/s | 138 kB 00:00 2023-07-20T18:48:01.554 INFO:teuthology.orchestra.run.smithi088.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 5.9 MB/s | 798 kB 00:00 2023-07-20T18:48:01.611 INFO:teuthology.orchestra.run.smithi057.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 601 kB/s | 110 kB 00:00 2023-07-20T18:48:01.703 INFO:teuthology.orchestra.run.smithi057.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 593 kB/s | 54 kB 00:00 2023-07-20T18:48:01.704 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-07-20T18:48:01.878 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T18:48:01.920 INFO:teuthology.orchestra.run.smithi057.stdout:(99/131): python3-scipy-1.0.0-21.module+el8.5.0 25 MB/s | 14 MB 00:00 2023-07-20T18:48:01.954 INFO:teuthology.orchestra.run.smithi057.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2023-07-20T18:48:02.070 INFO:teuthology.orchestra.run.smithi057.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2023-07-20T18:48:02.129 INFO:teuthology.orchestra.run.smithi057.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 221 kB/s | 39 kB 00:00 2023-07-20T18:48:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T18:48:02.196 INFO:teuthology.orchestra.run.smithi057.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 5.4 MB/s | 688 kB 00:00 2023-07-20T18:48:02.213 INFO:teuthology.orchestra.run.smithi088.stdout:(130/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 3.9 MB/s | 3.7 MB 00:00 2023-07-20T18:48:02.280 INFO:teuthology.orchestra.run.smithi057.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 3.5 MB/s | 538 kB 00:00 2023-07-20T18:48:02.306 INFO:teuthology.orchestra.run.smithi057.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2023-07-20T18:48:02.314 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:48:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:48:02.425 INFO:teuthology.orchestra.run.smithi057.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2023-07-20T18:48:02.473 INFO:teuthology.orchestra.run.smithi057.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 4.2 MB/s | 713 kB 00:00 2023-07-20T18:48:02.530 INFO:teuthology.orchestra.run.smithi088.stdout:(131/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 3.1 MB/s | 3.6 MB 00:01 2023-07-20T18:48:02.533 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:48:02.533 INFO:teuthology.orchestra.run.smithi088.stdout:Total 8.7 MB/s | 206 MB 00:23 2023-07-20T18:48:02.623 INFO:teuthology.orchestra.run.smithi057.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 177 kB/s | 27 kB 00:00 2023-07-20T18:48:02.680 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:48:02.723 INFO:teuthology.orchestra.run.smithi057.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2023-07-20T18:48:02.757 INFO:teuthology.orchestra.run.smithi057.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 2.3 MB/s | 320 kB 00:00 2023-07-20T18:48:02.790 INFO:teuthology.orchestra.run.smithi057.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 658 kB/s | 44 kB 00:00 2023-07-20T18:48:02.924 INFO:teuthology.orchestra.run.smithi057.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 3.4 MB/s | 457 kB 00:00 2023-07-20T18:48:03.049 INFO:teuthology.orchestra.run.smithi057.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 13 MB/s | 3.7 MB 00:00 2023-07-20T18:48:03.082 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:48:03.083 INFO:teuthology.orchestra.run.smithi057.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.4 MB/s | 225 kB 00:00 2023-07-20T18:48:03.175 INFO:teuthology.orchestra.run.smithi057.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 512 kB/s | 64 kB 00:00 2023-07-20T18:48:03.217 INFO:teuthology.orchestra.run.smithi057.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 948 kB/s | 126 kB 00:00 2023-07-20T18:48:03.276 INFO:teuthology.orchestra.run.smithi057.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 199 kB/s | 20 kB 00:00 2023-07-20T18:48:03.330 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:48:03.346 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-07-20T18:48:03.361 INFO:teuthology.orchestra.run.smithi057.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 719 kB/s | 103 kB 00:00 2023-07-20T18:48:03.409 INFO:teuthology.orchestra.run.smithi057.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 5.2 MB/s | 709 kB 00:00 2023-07-20T18:48:03.466 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-07-20T18:48:03.467 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-07-20T18:48:03.593 INFO:teuthology.orchestra.run.smithi057.stdout:(120/131): openblas-threads-0.3.15-3.el8.x86_64 21 MB/s | 4.9 MB 00:00 2023-07-20T18:48:03.618 INFO:teuthology.orchestra.run.smithi057.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 947 kB/s | 198 kB 00:00 2023-07-20T18:48:03.719 INFO:teuthology.orchestra.run.smithi057.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 2.0 MB/s | 202 kB 00:00 2023-07-20T18:48:03.728 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:48:03.802 INFO:teuthology.orchestra.run.smithi057.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 4.2 MB/s | 354 kB 00:00 2023-07-20T18:48:03.894 INFO:teuthology.orchestra.run.smithi057.stdout:(124/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-07-20T18:48:03.913 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:48:04.044 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:48:04.086 INFO:teuthology.orchestra.run.smithi057.stdout:(125/131): postgresql-server-10.21-2.module+el8 18 MB/s | 5.1 MB 00:00 2023-07-20T18:48:04.136 INFO:teuthology.orchestra.run.smithi057.stdout:(126/131): postgresql-contrib-10.21-2.module+el 3.3 MB/s | 808 kB 00:00 2023-07-20T18:48:04.212 INFO:teuthology.orchestra.run.smithi057.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 12 MB/s | 1.5 MB 00:00 2023-07-20T18:48:04.238 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:48:04.332 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:48:04.518 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:48:04.591 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-07-20T18:48:04.596 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-07-20T18:48:04.689 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T18:48:04.925 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T18:48:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T18:48:05.338 INFO:teuthology.orchestra.run.smithi057.stdout:(128/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8. 3.3 MB/s | 48 MB 00:14 2023-07-20T18:48:05.504 INFO:teuthology.orchestra.run.smithi057.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.7 MB/s | 798 kB 00:00 2023-07-20T18:48:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T18:48:05.663 INFO:teuthology.orchestra.run.smithi057.stdout:(130/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 2.4 MB/s | 3.7 MB 00:01 2023-07-20T18:48:05.775 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T18:48:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:48:06.021 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:48:06.188 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:48:06.205 INFO:teuthology.orchestra.run.smithi057.stdout:(131/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 1.8 MB/s | 3.6 MB 00:01 2023-07-20T18:48:06.209 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:48:06.210 INFO:teuthology.orchestra.run.smithi057.stdout:Total 8.9 MB/s | 206 MB 00:23 2023-07-20T18:48:06.323 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:48:06.533 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T18:48:06.736 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T18:48:06.929 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T18:48:07.054 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:48:07.494 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:48:07.638 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:48:07.886 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-07-20T18:48:08.032 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-07-20T18:48:08.032 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-07-20T18:48:08.256 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:48:08.479 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:48:09.078 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:48:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:48:09.590 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-07-20T18:48:09.676 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-07-20T18:48:09.681 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-07-20T18:48:09.752 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T18:48:09.910 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T18:48:09.966 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:48:10.087 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:48:10.190 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:48:10.236 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T18:48:10.420 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T18:48:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:48:10.656 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T18:48:10.831 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:48:10.852 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T18:48:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T18:48:11.045 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:48:11.200 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T18:48:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:48:11.439 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:48:11.603 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:48:11.713 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:48:11.919 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:48:12.017 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:48:12.166 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:48:12.345 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T18:48:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T18:48:13.039 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T18:48:13.200 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:48:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T18:48:13.640 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T18:48:13.794 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:48:13.899 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:48:14.036 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:48:14.166 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:48:14.194 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T18:48:14.398 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T18:48:14.544 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T18:48:14.566 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T18:48:14.729 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T18:48:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T18:48:14.861 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:48:15.000 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:48:15.108 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T18:48:15.138 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-07-20T18:48:15.144 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:48:15.357 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T18:48:15.519 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T18:48:15.572 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T18:48:15.748 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T18:48:15.755 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:48:15.767 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T18:48:15.919 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:48:15.920 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:48:15.957 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T18:48:16.077 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:48:16.119 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T18:48:16.264 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T18:48:16.328 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:48:16.437 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:48:16.497 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:48:16.511 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:48:16.702 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:48:16.759 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T18:48:16.835 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:48:16.991 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T18:48:17.052 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:48:17.187 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:48:17.250 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:48:17.392 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:48:17.458 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T18:48:17.560 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:48:17.616 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T18:48:17.651 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T18:48:17.744 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:48:17.802 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T18:48:17.840 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T18:48:17.907 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:48:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T18:48:18.070 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T18:48:18.114 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T18:48:18.144 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:48:18.242 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T18:48:18.249 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:48:18.273 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T18:48:18.380 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T18:48:18.408 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:48:18.484 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T18:48:18.614 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T18:48:18.625 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T18:48:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T18:48:18.811 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:48:18.898 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T18:48:19.066 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T18:48:19.142 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T18:48:19.356 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T18:48:19.559 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T18:48:19.631 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T18:48:19.791 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T18:48:19.858 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T18:48:19.952 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:48:20.106 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:48:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T18:48:20.268 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:48:20.430 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T18:48:20.440 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:48:20.568 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:48:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T18:48:20.710 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T18:48:20.879 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T18:48:20.895 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T18:48:21.099 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T18:48:21.105 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T18:48:21.235 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:48:21.290 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T18:48:21.397 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:48:21.473 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T18:48:21.557 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T18:48:21.569 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:48:21.779 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T18:48:21.856 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T18:48:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T18:48:22.129 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T18:48:22.308 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T18:48:22.312 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:48:22.333 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T18:48:22.452 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:48:22.529 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T18:48:22.579 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T18:48:22.695 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T18:48:22.736 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T18:48:22.881 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T18:48:22.914 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T18:48:23.032 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T18:48:23.166 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T18:48:23.184 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:48:23.228 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T18:48:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T18:48:23.327 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:48:23.459 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:48:23.468 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T18:48:23.524 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:48:23.657 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T18:48:23.735 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T18:48:23.806 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T18:48:23.979 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T18:48:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T18:48:24.118 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:48:24.194 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:48:24.209 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:48:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T18:48:24.405 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T18:48:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T18:48:24.501 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T18:48:24.563 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T18:48:24.581 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T18:48:24.708 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T18:48:24.724 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T18:48:24.725 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T18:48:24.877 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T18:48:25.010 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T18:48:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T18:48:25.037 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T18:48:25.177 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T18:48:25.217 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T18:48:25.275 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T18:48:25.318 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T18:48:25.407 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T18:48:25.502 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T18:48:25.584 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T18:48:25.589 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T18:48:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T18:48:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:48:25.963 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T18:48:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T18:48:26.218 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T18:48:26.252 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T18:48:26.435 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T18:48:26.488 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T18:48:26.688 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T18:48:26.930 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T18:48:27.233 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T18:48:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T18:48:27.553 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T18:48:27.677 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T18:48:27.865 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T18:48:27.904 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T18:48:27.973 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:48:28.070 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T18:48:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T18:48:28.228 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T18:48:28.533 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T18:48:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T18:48:28.772 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T18:48:28.831 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T18:48:29.013 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T18:48:29.058 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T18:48:29.173 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T18:48:29.260 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T18:48:29.416 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T18:48:29.566 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T18:48:29.602 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T18:48:29.822 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T18:48:29.838 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T18:48:29.859 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T18:48:29.986 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:48:30.057 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:48:30.084 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T18:48:30.115 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T18:48:30.253 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T18:48:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T18:48:30.308 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T18:48:30.402 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T18:48:30.493 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T18:48:30.582 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T18:48:30.676 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T18:48:30.760 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T18:48:30.902 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T18:48:30.934 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T18:48:31.053 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T18:48:31.119 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T18:48:31.205 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T18:48:31.297 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T18:48:31.417 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:48:31.510 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:48:31.520 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T18:48:31.753 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T18:48:31.758 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T18:48:31.955 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T18:48:31.975 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T18:48:32.134 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:48:32.213 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T18:48:32.239 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:48:32.440 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T18:48:32.497 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T18:48:32.639 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T18:48:32.640 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T18:48:32.802 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T18:48:32.842 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T18:48:32.971 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T18:48:33.137 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T18:48:33.316 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T18:48:33.507 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T18:48:33.708 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T18:48:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:48:33.942 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T18:48:34.130 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T18:48:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:48:34.230 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T18:48:34.459 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T18:48:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T18:48:34.676 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T18:48:34.765 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:48:34.774 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T18:48:34.927 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T18:48:35.039 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:48:35.070 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T18:48:35.109 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:48:35.263 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T18:48:35.296 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:48:35.297 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T18:48:35.621 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T18:48:35.853 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T18:48:36.026 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T18:48:36.061 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T18:48:36.233 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:48:36.283 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:48:36.363 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T18:48:36.458 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T18:48:36.542 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T18:48:36.724 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T18:48:37.069 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T18:48:37.395 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T18:48:37.661 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T18:48:37.913 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T18:48:38.077 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T18:48:38.301 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T18:48:38.597 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T18:48:38.826 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T18:48:38.964 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T18:48:39.138 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T18:48:39.334 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T18:48:39.518 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T18:48:39.724 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T18:48:39.904 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T18:48:40.032 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T18:48:40.085 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:48:40.299 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:48:40.308 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T18:48:40.531 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T18:48:40.765 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T18:48:40.883 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:48:41.070 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T18:48:41.180 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:48:41.258 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:48:41.318 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T18:48:41.407 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:48:41.495 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T18:48:41.756 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T18:48:43.174 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T18:48:43.465 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T18:48:43.766 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T18:48:44.133 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T18:48:44.409 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T18:48:44.635 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T18:48:45.527 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T18:48:45.708 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:48:45.803 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:48:46.025 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T18:48:51.481 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:48:51.934 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:48:52.669 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:48:53.003 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:48:53.148 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:48:53.291 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:48:54.218 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:48:54.218 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2023-07-20T18:48:54.219 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2023-07-20T18:48:54.219 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /dev 2023-07-20T18:48:54.219 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /run 2023-07-20T18:48:54.219 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2023-07-20T18:48:54.219 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2023-07-20T18:48:54.219 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2023-07-20T18:48:54.219 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2023-07-20T18:48:54.220 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /dev 2023-07-20T18:48:54.220 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:48:54.600 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:48:56.177 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:48:56.362 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:48:56.579 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:48:56.767 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:48:57.220 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:48:57.358 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:48:57.375 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:48:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:48:59.239 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:48:59.397 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T18:48:59.574 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T18:48:59.758 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:48:59.883 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:48:59.883 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2023-07-20T18:48:59.883 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2023-07-20T18:48:59.883 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2023-07-20T18:48:59.884 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /run 2023-07-20T18:48:59.884 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2023-07-20T18:48:59.884 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2023-07-20T18:48:59.884 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2023-07-20T18:48:59.884 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2023-07-20T18:48:59.884 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2023-07-20T18:48:59.885 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:48:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:49:00.270 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:49:00.311 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:49:01.272 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:49:01.441 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:49:01.531 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:49:01.847 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:49:01.923 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T18:49:02.020 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:49:02.236 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:49:02.341 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:49:02.352 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:49:02.428 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:49:02.880 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:49:03.028 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:49:03.044 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:49:03.646 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T18:49:03.823 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:49:03.857 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:49:04.041 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:49:04.083 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:49:04.233 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:49:04.584 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:49:04.713 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:49:04.758 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:49:04.886 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:49:04.992 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T18:49:05.208 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T18:49:05.361 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:49:05.577 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:49:05.850 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:49:06.809 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:49:06.993 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:49:07.084 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:49:07.452 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T18:49:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:49:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:49:07.877 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:49:07.888 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:49:09.167 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T18:49:09.306 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:49:09.336 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:49:09.370 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:49:09.502 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:49:09.543 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:49:09.694 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:49:10.045 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:49:10.183 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T18:49:10.218 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:49:10.530 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T18:49:10.726 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T18:49:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T18:49:11.182 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:49:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:49:11.612 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T18:49:11.786 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T18:49:11.983 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T18:49:14.160 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:49:14.440 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:49:14.458 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:49:14.543 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:49:14.543 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /sys 2023-07-20T18:49:14.543 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /proc 2023-07-20T18:49:14.544 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /dev 2023-07-20T18:49:14.544 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /run 2023-07-20T18:49:14.544 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /mnt 2023-07-20T18:49:14.544 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /var/tmp 2023-07-20T18:49:14.544 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /home 2023-07-20T18:49:14.544 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /tmp 2023-07-20T18:49:14.545 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /dev 2023-07-20T18:49:14.545 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:49:14.646 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T18:49:14.766 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:49:14.947 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:49:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T18:49:15.239 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T18:49:15.640 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T18:49:15.654 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T18:49:15.839 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:49:15.938 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:49:15.956 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T18:49:16.173 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:49:16.178 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T18:49:16.186 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:49:16.331 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:49:16.339 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:49:16.424 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T18:49:16.467 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T18:49:16.467 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:49:16.567 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:49:16.701 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:49:16.701 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:49:16.758 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:49:16.807 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:49:16.946 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:49:17.005 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T18:49:17.019 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:49:17.155 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:49:17.193 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T18:49:17.302 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:49:17.385 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T18:49:17.522 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:49:17.650 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:49:17.651 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T18:49:17.651 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T18:49:17.651 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T18:49:17.651 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T18:49:17.651 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T18:49:17.651 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:49:17.651 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:49:17.652 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:49:17.652 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T18:49:17.652 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T18:49:17.652 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T18:49:17.652 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T18:49:17.652 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T18:49:17.652 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T18:49:17.653 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T18:49:17.653 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T18:49:17.653 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T18:49:17.653 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T18:49:17.653 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T18:49:17.653 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T18:49:17.653 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T18:49:17.654 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T18:49:17.654 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T18:49:17.654 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T18:49:17.654 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T18:49:17.654 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T18:49:17.654 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T18:49:17.654 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T18:49:17.654 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T18:49:17.655 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T18:49:17.655 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T18:49:17.655 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T18:49:17.655 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T18:49:17.655 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T18:49:17.656 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T18:49:17.656 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T18:49:17.656 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T18:49:17.656 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T18:49:17.656 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T18:49:17.657 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T18:49:17.657 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T18:49:17.657 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T18:49:17.657 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T18:49:17.657 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T18:49:17.657 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T18:49:17.657 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T18:49:17.658 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T18:49:17.658 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T18:49:17.658 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T18:49:17.658 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T18:49:17.658 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T18:49:17.658 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T18:49:17.658 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T18:49:17.659 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T18:49:17.659 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T18:49:17.659 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T18:49:17.659 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T18:49:17.659 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T18:49:17.659 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T18:49:17.660 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T18:49:17.660 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T18:49:17.660 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T18:49:17.660 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T18:49:17.660 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T18:49:17.661 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T18:49:17.661 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T18:49:17.661 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T18:49:17.661 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T18:49:17.661 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T18:49:17.661 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T18:49:17.662 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T18:49:17.662 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T18:49:17.662 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T18:49:17.662 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T18:49:17.662 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T18:49:17.662 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T18:49:17.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T18:49:17.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T18:49:17.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T18:49:17.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T18:49:17.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T18:49:17.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T18:49:17.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T18:49:17.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T18:49:17.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T18:49:17.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T18:49:17.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T18:49:17.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T18:49:17.668 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T18:49:17.668 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T18:49:17.669 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T18:49:17.669 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T18:49:17.669 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T18:49:17.669 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T18:49:17.669 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T18:49:17.670 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T18:49:17.670 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T18:49:17.670 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T18:49:17.670 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T18:49:17.670 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T18:49:17.670 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T18:49:17.671 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T18:49:17.671 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T18:49:17.671 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T18:49:17.671 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T18:49:17.671 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T18:49:17.671 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T18:49:17.671 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T18:49:17.672 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T18:49:17.672 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T18:49:17.672 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T18:49:17.672 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T18:49:17.672 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T18:49:17.673 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T18:49:17.673 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T18:49:17.673 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T18:49:17.673 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T18:49:17.673 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T18:49:17.673 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T18:49:17.673 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T18:49:17.674 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T18:49:17.674 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T18:49:17.674 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T18:49:17.674 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T18:49:17.675 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T18:49:17.675 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T18:49:17.675 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T18:49:17.675 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T18:49:17.675 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T18:49:17.675 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T18:49:17.675 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T18:49:17.676 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:49:17.676 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T18:49:18.084 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:49:18.288 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:49:18.332 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:49:19.516 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:49:19.791 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:49:19.812 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:49:20.001 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T18:49:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:49:20.143 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:54 ago on Thu 20 Jul 2023 06:47:26 PM UTC. 2023-07-20T18:49:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T18:49:20.517 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T18:49:20.827 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T18:49:21.032 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:49:21.107 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:49:21.114 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:49:21.270 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:49:21.316 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:49:21.328 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:49:21.421 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T18:49:21.474 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:49:21.482 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:49:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T18:49:21.602 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:49:21.619 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T18:49:21.811 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:49:21.811 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:49:21.812 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:49:22.055 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:49:22.086 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:49:22.258 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:49:22.267 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:49:22.808 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:49:22.808 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T18:49:22.808 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T18:49:22.808 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T18:49:22.809 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T18:49:22.809 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T18:49:22.809 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:49:22.809 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:49:22.809 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:49:22.809 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T18:49:22.809 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T18:49:22.809 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T18:49:22.810 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T18:49:22.810 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T18:49:22.810 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T18:49:22.810 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T18:49:22.810 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T18:49:22.810 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T18:49:22.819 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T18:49:22.819 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T18:49:22.819 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T18:49:22.819 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T18:49:22.820 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T18:49:22.820 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T18:49:22.820 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T18:49:22.820 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T18:49:22.820 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T18:49:22.820 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T18:49:22.820 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T18:49:22.821 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T18:49:22.821 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T18:49:22.821 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T18:49:22.821 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T18:49:22.821 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T18:49:22.821 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T18:49:22.821 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T18:49:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T18:49:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T18:49:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T18:49:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T18:49:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T18:49:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T18:49:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T18:49:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T18:49:22.823 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T18:49:22.823 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T18:49:22.823 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T18:49:22.823 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T18:49:22.824 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T18:49:22.824 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T18:49:22.824 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T18:49:22.824 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T18:49:22.824 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T18:49:22.824 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T18:49:22.824 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T18:49:22.825 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T18:49:22.825 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T18:49:22.825 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T18:49:22.825 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T18:49:22.825 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T18:49:22.825 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T18:49:22.825 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T18:49:22.825 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T18:49:22.826 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T18:49:22.826 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T18:49:22.826 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T18:49:22.826 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T18:49:22.826 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T18:49:22.826 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T18:49:22.826 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T18:49:22.827 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T18:49:22.827 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T18:49:22.827 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T18:49:22.827 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T18:49:22.827 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T18:49:22.827 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T18:49:22.827 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T18:49:22.827 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T18:49:22.828 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T18:49:22.828 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T18:49:22.828 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T18:49:22.828 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T18:49:22.828 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T18:49:22.828 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T18:49:22.828 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T18:49:22.829 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T18:49:22.829 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T18:49:22.829 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T18:49:22.829 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T18:49:22.829 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T18:49:22.829 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T18:49:22.829 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T18:49:22.829 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T18:49:22.830 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T18:49:22.830 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T18:49:22.830 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T18:49:22.830 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T18:49:22.830 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T18:49:22.830 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T18:49:22.830 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T18:49:22.830 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T18:49:22.831 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T18:49:22.831 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T18:49:22.831 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T18:49:22.831 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T18:49:22.831 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T18:49:22.831 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T18:49:22.831 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T18:49:22.832 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T18:49:22.832 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T18:49:22.832 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T18:49:22.832 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T18:49:22.832 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T18:49:22.832 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T18:49:22.832 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T18:49:22.832 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T18:49:22.833 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T18:49:22.833 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T18:49:22.833 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T18:49:22.833 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T18:49:22.833 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T18:49:22.833 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T18:49:22.833 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T18:49:22.834 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T18:49:22.834 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T18:49:22.834 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T18:49:22.834 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T18:49:22.834 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T18:49:22.834 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T18:49:22.834 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T18:49:22.834 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T18:49:22.835 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T18:49:22.835 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:49:22.835 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T18:49:23.459 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:49:23.629 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:49:23.764 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:49:24.161 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T18:49:24.422 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:49:24.489 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:49:24.645 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:49:24.674 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:49:25.279 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:49:25.280 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:59 ago on Thu 20 Jul 2023 06:47:26 PM UTC. 2023-07-20T18:49:26.205 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T18:49:26.444 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:49:26.508 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:49:26.662 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:49:26.744 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:49:26.904 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:49:27.347 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:49:27.513 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:49:31.661 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-07-20T18:49:31.877 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:49:31.877 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2023-07-20T18:49:31.878 INFO:teuthology.orchestra.run.smithi192.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T18:49:31.878 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.878 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.878 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-07-20T18:49:31.878 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T18:49:31.878 INFO:teuthology.orchestra.run.smithi192.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T18:49:31.878 INFO:teuthology.orchestra.run.smithi192.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T18:49:31.879 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.879 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.879 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.879 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.879 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:31.879 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.879 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.879 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.880 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:31.880 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:31.880 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:31.880 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:31.880 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:31.880 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.881 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.881 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:31.881 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.881 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.881 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.882 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:31.882 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:31.882 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.882 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:31.882 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T18:49:31.882 INFO:teuthology.orchestra.run.smithi192.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T18:49:31.882 INFO:teuthology.orchestra.run.smithi192.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T18:49:31.883 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T18:49:31.883 INFO:teuthology.orchestra.run.smithi192.stdout: jq-1.6-3.el8.x86_64 2023-07-20T18:49:31.883 INFO:teuthology.orchestra.run.smithi192.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T18:49:31.883 INFO:teuthology.orchestra.run.smithi192.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T18:49:31.883 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T18:49:31.883 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.883 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.884 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.884 INFO:teuthology.orchestra.run.smithi192.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T18:49:31.884 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:49:31.884 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T18:49:31.884 INFO:teuthology.orchestra.run.smithi192.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T18:49:31.884 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:49:31.884 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T18:49:31.885 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.885 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.885 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T18:49:31.885 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.885 INFO:teuthology.orchestra.run.smithi192.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T18:49:31.885 INFO:teuthology.orchestra.run.smithi192.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T18:49:31.885 INFO:teuthology.orchestra.run.smithi192.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T18:49:31.885 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T18:49:31.886 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T18:49:31.886 INFO:teuthology.orchestra.run.smithi192.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T18:49:31.886 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T18:49:31.886 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T18:49:31.886 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T18:49:31.886 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T18:49:31.886 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T18:49:31.887 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T18:49:31.887 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T18:49:31.887 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T18:49:31.887 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:49:31.887 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:49:31.887 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:49:31.887 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T18:49:31.888 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T18:49:31.888 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T18:49:31.888 INFO:teuthology.orchestra.run.smithi192.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T18:49:31.888 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T18:49:31.888 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.888 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.888 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.889 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T18:49:31.889 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T18:49:31.889 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T18:49:31.889 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T18:49:31.889 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T18:49:31.889 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T18:49:31.889 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T18:49:31.889 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T18:49:31.890 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T18:49:31.890 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T18:49:31.890 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T18:49:31.890 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T18:49:31.890 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T18:49:31.890 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T18:49:31.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T18:49:31.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T18:49:31.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T18:49:31.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T18:49:31.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T18:49:31.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T18:49:31.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T18:49:31.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T18:49:31.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T18:49:31.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T18:49:31.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T18:49:31.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T18:49:31.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T18:49:31.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T18:49:31.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T18:49:31.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T18:49:31.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T18:49:31.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T18:49:31.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T18:49:31.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T18:49:31.896 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T18:49:31.896 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T18:49:31.896 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T18:49:31.896 INFO:teuthology.orchestra.run.smithi192.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T18:49:31.896 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T18:49:31.896 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T18:49:31.897 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T18:49:31.897 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T18:49:31.897 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T18:49:31.897 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T18:49:31.897 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.897 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.897 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:31.897 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T18:49:31.898 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T18:49:31.898 INFO:teuthology.orchestra.run.smithi192.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T18:49:31.898 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T18:49:31.898 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T18:49:31.899 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T18:49:31.900 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:49:31.900 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-07-20T18:49:32.060 DEBUG:teuthology.parallel:result is None 2023-07-20T18:49:32.906 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:49:34.250 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T18:49:34.604 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T18:49:34.776 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T18:49:35.039 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T18:49:35.194 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:49:35.461 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:49:35.700 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T18:49:35.875 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T18:49:36.037 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T18:49:36.228 INFO:teuthology.orchestra.run.smithi088.stdout:Installed products updated. 2023-07-20T18:49:36.447 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:49:36.448 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2023-07-20T18:49:36.448 INFO:teuthology.orchestra.run.smithi088.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T18:49:36.448 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.449 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.450 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-07-20T18:49:36.450 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T18:49:36.450 INFO:teuthology.orchestra.run.smithi088.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T18:49:36.450 INFO:teuthology.orchestra.run.smithi088.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T18:49:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.452 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.452 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:36.452 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.452 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.453 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.453 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:36.453 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:36.454 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:36.454 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:36.454 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:36.454 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.454 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.454 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:36.455 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.455 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.455 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.456 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:36.456 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:36.456 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.456 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:36.457 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T18:49:36.457 INFO:teuthology.orchestra.run.smithi088.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T18:49:36.457 INFO:teuthology.orchestra.run.smithi088.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T18:49:36.457 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T18:49:36.457 INFO:teuthology.orchestra.run.smithi088.stdout: jq-1.6-3.el8.x86_64 2023-07-20T18:49:36.458 INFO:teuthology.orchestra.run.smithi088.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T18:49:36.458 INFO:teuthology.orchestra.run.smithi088.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T18:49:36.458 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T18:49:36.458 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.458 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.458 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.458 INFO:teuthology.orchestra.run.smithi088.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T18:49:36.459 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:49:36.459 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T18:49:36.460 INFO:teuthology.orchestra.run.smithi088.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T18:49:36.460 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:49:36.460 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T18:49:36.460 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.460 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.461 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T18:49:36.461 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.461 INFO:teuthology.orchestra.run.smithi088.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T18:49:36.461 INFO:teuthology.orchestra.run.smithi088.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T18:49:36.461 INFO:teuthology.orchestra.run.smithi088.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T18:49:36.461 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T18:49:36.461 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T18:49:36.461 INFO:teuthology.orchestra.run.smithi088.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T18:49:36.462 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T18:49:36.462 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T18:49:36.462 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T18:49:36.462 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T18:49:36.462 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T18:49:36.462 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T18:49:36.462 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T18:49:36.463 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T18:49:36.463 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:49:36.463 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:49:36.463 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:49:36.463 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T18:49:36.463 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T18:49:36.463 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T18:49:36.463 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T18:49:36.464 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T18:49:36.464 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.464 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.464 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.464 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T18:49:36.464 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T18:49:36.465 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T18:49:36.465 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T18:49:36.465 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T18:49:36.465 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T18:49:36.465 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T18:49:36.465 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T18:49:36.465 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T18:49:36.466 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T18:49:36.466 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T18:49:36.466 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T18:49:36.466 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T18:49:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T18:49:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T18:49:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T18:49:36.467 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T18:49:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T18:49:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T18:49:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T18:49:36.468 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T18:49:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T18:49:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T18:49:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T18:49:36.469 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T18:49:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T18:49:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T18:49:36.470 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T18:49:36.471 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T18:49:36.471 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.471 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.471 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T18:49:36.472 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T18:49:36.472 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.472 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T18:49:36.473 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T18:49:36.473 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T18:49:36.473 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T18:49:36.473 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T18:49:36.474 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T18:49:36.474 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T18:49:36.474 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T18:49:36.475 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T18:49:36.475 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T18:49:36.475 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T18:49:36.475 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T18:49:36.475 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T18:49:36.476 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.476 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.476 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:36.477 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T18:49:36.477 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T18:49:36.477 INFO:teuthology.orchestra.run.smithi088.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T18:49:36.477 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T18:49:36.477 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T18:49:36.478 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T18:49:36.478 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-20T18:49:36.479 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-20T18:49:36.636 DEBUG:teuthology.parallel:result is None 2023-07-20T18:49:39.450 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:49:39.694 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:49:39.733 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:49:40.038 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T18:49:40.231 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T18:49:40.580 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T18:49:40.968 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T18:49:41.160 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:49:41.298 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:49:41.519 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:49:41.547 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:49:41.719 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:49:41.742 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:49:41.914 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T18:49:41.914 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:49:42.134 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:49:42.134 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:49:42.436 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:49:42.645 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:49:42.947 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:49:42.947 INFO:teuthology.orchestra.run.smithi057.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-07-20T18:49:42.948 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:49:43.588 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T18:49:43.589 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T18:49:43.589 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T18:49:43.589 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T18:49:43.589 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T18:49:43.589 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:49:43.590 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:49:43.590 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:49:43.590 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T18:49:43.590 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T18:49:43.590 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T18:49:43.590 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T18:49:43.590 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T18:49:43.590 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T18:49:43.591 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T18:49:43.591 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T18:49:43.591 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T18:49:43.592 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T18:49:43.592 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T18:49:43.592 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T18:49:43.592 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T18:49:43.592 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T18:49:43.593 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T18:49:43.593 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T18:49:43.593 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T18:49:43.593 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T18:49:43.594 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T18:49:43.594 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T18:49:43.594 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T18:49:43.594 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T18:49:43.594 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T18:49:43.595 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T18:49:43.595 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T18:49:43.595 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T18:49:43.595 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T18:49:43.595 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T18:49:43.595 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T18:49:43.595 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T18:49:43.596 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T18:49:43.596 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T18:49:43.596 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T18:49:43.596 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T18:49:43.596 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T18:49:43.596 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T18:49:43.597 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T18:49:43.597 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T18:49:43.597 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T18:49:43.597 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T18:49:43.597 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T18:49:43.597 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T18:49:43.597 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T18:49:43.597 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T18:49:43.598 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T18:49:43.598 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T18:49:43.598 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T18:49:43.598 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T18:49:43.598 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T18:49:43.598 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T18:49:43.598 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T18:49:43.598 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T18:49:43.599 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T18:49:43.599 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T18:49:43.599 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T18:49:43.599 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T18:49:43.599 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T18:49:43.599 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T18:49:43.599 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T18:49:43.600 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T18:49:43.600 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T18:49:43.600 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T18:49:43.600 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T18:49:43.600 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T18:49:43.600 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T18:49:43.600 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T18:49:43.601 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T18:49:43.601 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T18:49:43.601 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T18:49:43.601 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T18:49:43.601 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T18:49:43.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T18:49:43.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T18:49:43.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T18:49:43.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T18:49:43.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T18:49:43.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T18:49:43.603 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T18:49:43.603 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T18:49:43.603 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T18:49:43.603 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T18:49:43.603 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T18:49:43.603 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T18:49:43.603 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T18:49:43.603 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T18:49:43.604 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T18:49:43.604 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T18:49:43.604 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T18:49:43.604 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T18:49:43.604 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T18:49:43.604 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T18:49:43.604 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T18:49:43.605 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T18:49:43.605 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T18:49:43.605 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T18:49:43.605 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T18:49:43.605 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T18:49:43.605 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T18:49:43.605 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T18:49:43.606 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T18:49:43.606 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T18:49:43.606 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T18:49:43.606 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T18:49:43.606 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T18:49:43.606 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T18:49:43.606 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T18:49:43.606 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T18:49:43.607 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T18:49:43.607 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T18:49:43.607 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T18:49:43.607 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T18:49:43.607 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T18:49:43.607 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T18:49:43.607 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T18:49:43.607 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T18:49:43.608 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T18:49:43.608 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T18:49:43.608 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T18:49:43.608 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T18:49:43.608 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T18:49:43.608 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T18:49:43.608 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T18:49:43.609 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T18:49:43.609 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:49:43.609 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T18:49:46.284 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:49:46.285 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:17 ago on Thu 20 Jul 2023 06:47:29 PM UTC. 2023-07-20T18:49:53.623 INFO:teuthology.orchestra.run.smithi057.stdout:Installed products updated. 2023-07-20T18:49:53.859 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:49:53.860 INFO:teuthology.orchestra.run.smithi057.stdout:Upgraded: 2023-07-20T18:49:53.860 INFO:teuthology.orchestra.run.smithi057.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T18:49:53.860 INFO:teuthology.orchestra.run.smithi057.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.860 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.860 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-07-20T18:49:53.861 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T18:49:53.861 INFO:teuthology.orchestra.run.smithi057.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T18:49:53.861 INFO:teuthology.orchestra.run.smithi057.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T18:49:53.861 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.861 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.861 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.861 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.862 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:53.862 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.862 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.862 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.862 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:53.862 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:53.862 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:53.862 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:53.863 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:53.863 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.863 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.863 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:53.863 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.863 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.863 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.864 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:53.864 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:53.864 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.864 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:49:53.864 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T18:49:53.864 INFO:teuthology.orchestra.run.smithi057.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T18:49:53.864 INFO:teuthology.orchestra.run.smithi057.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T18:49:53.864 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T18:49:53.865 INFO:teuthology.orchestra.run.smithi057.stdout: jq-1.6-3.el8.x86_64 2023-07-20T18:49:53.865 INFO:teuthology.orchestra.run.smithi057.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T18:49:53.865 INFO:teuthology.orchestra.run.smithi057.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T18:49:53.865 INFO:teuthology.orchestra.run.smithi057.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T18:49:53.865 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.865 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.865 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.866 INFO:teuthology.orchestra.run.smithi057.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T18:49:53.866 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:49:53.866 INFO:teuthology.orchestra.run.smithi057.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T18:49:53.866 INFO:teuthology.orchestra.run.smithi057.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T18:49:53.866 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:49:53.866 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T18:49:53.866 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.866 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.867 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T18:49:53.867 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.867 INFO:teuthology.orchestra.run.smithi057.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T18:49:53.867 INFO:teuthology.orchestra.run.smithi057.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T18:49:53.867 INFO:teuthology.orchestra.run.smithi057.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T18:49:53.867 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T18:49:53.867 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T18:49:53.868 INFO:teuthology.orchestra.run.smithi057.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T18:49:53.868 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T18:49:53.868 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T18:49:53.868 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T18:49:53.868 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T18:49:53.868 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T18:49:53.868 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T18:49:53.868 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T18:49:53.869 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T18:49:53.869 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:49:53.869 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:49:53.869 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:49:53.869 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T18:49:53.869 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T18:49:53.869 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T18:49:53.870 INFO:teuthology.orchestra.run.smithi057.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T18:49:53.870 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T18:49:53.870 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.870 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.870 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.870 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T18:49:53.870 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T18:49:53.870 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T18:49:53.871 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T18:49:53.871 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T18:49:53.871 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T18:49:53.871 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T18:49:53.871 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T18:49:53.871 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T18:49:53.871 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T18:49:53.871 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T18:49:53.872 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T18:49:53.872 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T18:49:53.872 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T18:49:53.872 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T18:49:53.872 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T18:49:53.872 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T18:49:53.872 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T18:49:53.873 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T18:49:53.873 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T18:49:53.873 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T18:49:53.873 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T18:49:53.873 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T18:49:53.873 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T18:49:53.873 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T18:49:53.873 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T18:49:53.874 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T18:49:53.874 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T18:49:53.874 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T18:49:53.874 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.874 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.875 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T18:49:53.875 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T18:49:53.875 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.875 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T18:49:53.875 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T18:49:53.875 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T18:49:53.875 INFO:teuthology.orchestra.run.smithi057.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T18:49:53.875 INFO:teuthology.orchestra.run.smithi057.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T18:49:53.876 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T18:49:53.876 INFO:teuthology.orchestra.run.smithi057.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T18:49:53.876 INFO:teuthology.orchestra.run.smithi057.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T18:49:53.876 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T18:49:53.876 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T18:49:53.876 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T18:49:53.876 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T18:49:53.876 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T18:49:53.877 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.877 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.877 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:49:53.877 INFO:teuthology.orchestra.run.smithi057.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T18:49:53.877 INFO:teuthology.orchestra.run.smithi057.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T18:49:53.877 INFO:teuthology.orchestra.run.smithi057.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T18:49:53.877 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T18:49:53.878 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T18:49:53.878 INFO:teuthology.orchestra.run.smithi057.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T18:49:53.878 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:49:53.878 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-07-20T18:49:54.428 DEBUG:teuthology.parallel:result is None 2023-07-20T18:49:54.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:49:54.720 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T18:49:54.775 INFO:teuthology.orchestra.run.smithi057.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:49:54.777 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:49:54.777 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T18:49:54.778 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:49:55.014 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T18:49:55.042 INFO:teuthology.orchestra.run.smithi088.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:49:55.043 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:49:55.043 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T18:49:55.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:49:55.265 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T18:49:55.295 INFO:teuthology.orchestra.run.smithi192.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:49:55.295 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:49:55.296 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T18:49:55.297 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-20T18:49:55.297 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:49:55.297 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T18:49:55.390 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:49:55.390 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T18:49:55.438 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:49:55.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T18:49:55.486 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-20T18:49:55.486 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:49:55.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T18:49:55.555 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T18:49:55.664 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:49:55.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T18:49:55.697 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T18:49:55.766 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:49:55.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T18:49:55.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T18:49:55.871 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-20T18:49:55.872 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:49:55.872 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T18:49:55.941 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T18:49:56.048 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:49:56.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T18:49:56.080 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T18:49:56.150 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:49:56.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T18:49:56.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T18:49:56.254 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-20T18:49:56.255 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:49:56.255 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T18:49:56.323 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T18:49:56.433 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:49:56.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T18:49:56.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T18:49:56.536 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:49:56.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T18:49:56.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T18:49:56.639 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-20T18:49:56.730 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client dirsize rbytes': False, 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'subvols': {'create': 2}} 2023-07-20T18:49:56.731 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:49:56.741 INFO:tasks.cephadm:Cluster fsid is 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:49:56.742 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-20T18:49:56.743 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.57', 'mon.b': '172.21.15.88', 'mon.c': '172.21.15.192'} 2023-07-20T18:49:56.743 INFO:tasks.cephadm:First mon is mon.a on smithi057 2023-07-20T18:49:56.743 INFO:tasks.cephadm:First mgr is x 2023-07-20T18:49:56.743 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-20T18:49:56.744 DEBUG:teuthology.orchestra.run.smithi057:> sudo hostname $(hostname -s) 2023-07-20T18:49:56.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname $(hostname -s) 2023-07-20T18:49:56.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2023-07-20T18:49:56.879 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-20T18:49:56.880 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:50:02.135 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71562/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-5040-g1b0b2eb0', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-5040.g1b0b2eb0'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-19 22:37:13.930734', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230719.201220', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-07-20T18:50:07.398 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-pdonnell-testing-20230719.201220, sha1 1b0b2eb005a7c51d97a788b9900d511e1394b537 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:50:07.399 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm 2023-07-20T18:50:07.399 DEBUG:teuthology.orchestra.run.smithi057:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T18:50:08.180 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 18:50 /home/ubuntu/cephtest/cephadm 2023-07-20T18:50:08.181 DEBUG:teuthology.orchestra.run.smithi088:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T18:50:08.900 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 18:50 /home/ubuntu/cephtest/cephadm 2023-07-20T18:50:08.901 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T18:50:09.790 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 18:50 /home/ubuntu/cephtest/cephadm 2023-07-20T18:50:09.791 DEBUG:teuthology.orchestra.run.smithi057:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T18:50:09.842 DEBUG:teuthology.orchestra.run.smithi088:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T18:50:09.865 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T18:50:09.903 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 on all hosts... 2023-07-20T18:50:09.903 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T18:50:09.911 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T18:50:09.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T18:50:11.188 INFO:teuthology.orchestra.run.smithi088.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T18:50:11.213 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T18:50:11.943 INFO:teuthology.orchestra.run.smithi057.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T18:50:37.269 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-20T18:50:37.269 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T18:50:37.269 INFO:teuthology.orchestra.run.smithi088.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T18:50:37.270 INFO:teuthology.orchestra.run.smithi088.stdout: "repo_digests": [ 2023-07-20T18:50:37.270 INFO:teuthology.orchestra.run.smithi088.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T18:50:37.270 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-07-20T18:50:37.270 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-20T18:50:42.635 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2023-07-20T18:50:42.636 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T18:50:42.636 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T18:50:42.636 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2023-07-20T18:50:42.636 INFO:teuthology.orchestra.run.smithi192.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T18:50:42.636 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2023-07-20T18:50:42.637 INFO:teuthology.orchestra.run.smithi192.stdout:} 2023-07-20T18:50:50.228 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-07-20T18:50:50.229 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T18:50:50.229 INFO:teuthology.orchestra.run.smithi057.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T18:50:50.229 INFO:teuthology.orchestra.run.smithi057.stdout: "repo_digests": [ 2023-07-20T18:50:50.229 INFO:teuthology.orchestra.run.smithi057.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T18:50:50.229 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-07-20T18:50:50.230 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-07-20T18:50:50.250 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph 2023-07-20T18:50:50.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph 2023-07-20T18:50:50.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2023-07-20T18:50:50.421 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 777 /etc/ceph 2023-07-20T18:50:50.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /etc/ceph 2023-07-20T18:50:50.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2023-07-20T18:50:50.561 INFO:tasks.cephadm:Writing seed config... 2023-07-20T18:50:50.562 INFO:tasks.cephadm: override: [client] client dirsize rbytes = False 2023-07-20T18:50:50.562 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-20T18:50:50.563 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-20T18:50:50.563 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-20T18:50:50.563 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-20T18:50:50.563 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-20T18:50:50.564 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-07-20T18:50:50.564 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-20T18:50:50.564 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-20T18:50:50.564 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-20T18:50:50.564 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-20T18:50:50.565 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-20T18:50:50.565 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-20T18:50:50.565 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-20T18:50:50.565 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-20T18:50:50.566 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-20T18:50:50.566 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-20T18:50:50.567 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-20T18:50:50.567 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-20T18:50:50.567 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-20T18:50:50.568 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-20T18:50:50.568 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-20T18:50:50.568 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-20T18:50:50.568 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-20T18:50:50.569 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-20T18:50:50.569 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-20T18:50:50.569 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-20T18:50:50.569 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-20T18:50:50.570 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-20T18:50:50.570 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-07-20T18:50:50.570 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-20T18:50:50.570 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-07-20T18:50:50.570 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-07-20T18:50:50.571 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-20T18:50:50.571 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-20T18:50:50.571 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-07-20T18:50:50.571 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-20T18:50:50.571 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-20T18:50:50.572 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-20T18:50:50.572 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-20T18:50:50.572 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-20T18:50:50.572 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-20T18:50:50.573 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-20T18:50:50.573 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-07-20T18:50:50.574 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:50:50.574 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-20T18:50:50.624 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 38a58c3c-272e-11ee-9b34-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client dirsize rbytes = False client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-20T18:50:50.625 DEBUG:teuthology.orchestra.run.smithi057:mon.a> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.a.service 2023-07-20T18:50:50.674 DEBUG:teuthology.orchestra.run.smithi057:mgr.x> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.x.service 2023-07-20T18:50:50.721 INFO:tasks.cephadm:Bootstrapping... 2023-07-20T18:50:50.722 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 -v bootstrap --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.57 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-20T18:50:50.742 INFO:journalctl@ceph.mon.a.smithi057.stdout:-- Logs begin at Thu 2023-07-20 18:30:45 UTC. -- 2023-07-20T18:50:50.792 INFO:journalctl@ceph.mgr.x.smithi057.stdout:-- Logs begin at Thu 2023-07-20 18:30:45 UTC. -- 2023-07-20T18:50:50.951 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:50:50.952 INFO:teuthology.orchestra.run.smithi057.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537', '-v', 'bootstrap', '--fsid', '38a58c3c-272e-11ee-9b34-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.57', '--skip-admin-label'] 2023-07-20T18:50:51.124 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stdout 4.1.1 2023-07-20T18:50:51.125 INFO:teuthology.orchestra.run.smithi057.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-20T18:50:51.125 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying podman|docker is present... 2023-07-20T18:50:51.295 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stdout 4.1.1 2023-07-20T18:50:51.295 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying lvm2 is present... 2023-07-20T18:50:51.296 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying time synchronization is in place... 2023-07-20T18:50:51.310 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T18:50:51.311 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T18:50:51.323 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T18:50:51.324 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2023-07-20T18:50:51.338 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout enabled 2023-07-20T18:50:51.350 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout active 2023-07-20T18:50:51.350 INFO:teuthology.orchestra.run.smithi057.stdout:Unit chronyd.service is enabled and running 2023-07-20T18:50:51.351 INFO:teuthology.orchestra.run.smithi057.stdout:Repeating the final host check... 2023-07-20T18:50:51.521 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stdout 4.1.1 2023-07-20T18:50:51.521 INFO:teuthology.orchestra.run.smithi057.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-20T18:50:51.521 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl is present 2023-07-20T18:50:51.521 INFO:teuthology.orchestra.run.smithi057.stdout:lvcreate is present 2023-07-20T18:50:51.533 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T18:50:51.536 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T18:50:51.546 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T18:50:51.546 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2023-07-20T18:50:51.560 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout enabled 2023-07-20T18:50:51.572 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout active 2023-07-20T18:50:51.577 INFO:teuthology.orchestra.run.smithi057.stdout:Unit chronyd.service is enabled and running 2023-07-20T18:50:51.578 INFO:teuthology.orchestra.run.smithi057.stdout:Host looks OK 2023-07-20T18:50:51.578 INFO:teuthology.orchestra.run.smithi057.stdout:Cluster fsid: 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:50:51.578 INFO:teuthology.orchestra.run.smithi057.stdout:Acquiring lock 140145521994216 on /run/cephadm/38a58c3c-272e-11ee-9b34-001a4aab830c.lock 2023-07-20T18:50:51.578 INFO:teuthology.orchestra.run.smithi057.stdout:Lock 140145521994216 acquired on /run/cephadm/38a58c3c-272e-11ee-9b34-001a4aab830c.lock 2023-07-20T18:50:51.579 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying IP 172.21.15.57 port 3300 ... 2023-07-20T18:50:51.579 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying IP 172.21.15.57 port 6789 ... 2023-07-20T18:50:51.579 INFO:teuthology.orchestra.run.smithi057.stdout:Base mon IP(s) is [172.21.15.57:3300, 172.21.15.57:6789], mon addrv is [v2:172.21.15.57:3300,v1:172.21.15.57:6789] 2023-07-20T18:50:51.581 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-07-20T18:50:51.582 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.57 metric 100 2023-07-20T18:50:51.589 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-20T18:50:51.589 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-07-20T18:50:51.589 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2023-07-20T18:50:51.599 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-20T18:50:51.599 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-20T18:50:51.599 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T18:50:51.599 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-07-20T18:50:51.600 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1562/64 scope link 2023-07-20T18:50:51.600 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T18:50:51.600 INFO:teuthology.orchestra.run.smithi057.stdout:Mon IP `172.21.15.57` is in CIDR network `172.21.0.0/20` 2023-07-20T18:50:51.600 INFO:teuthology.orchestra.run.smithi057.stdout:Mon IP `172.21.15.57` is in CIDR network `172.21.0.0/20` 2023-07-20T18:50:51.600 INFO:teuthology.orchestra.run.smithi057.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-20T18:50:51.600 INFO:teuthology.orchestra.run.smithi057.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-20T18:50:51.601 INFO:teuthology.orchestra.run.smithi057.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T18:50:52.075 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stdout 3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T18:50:52.076 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T18:50:52.076 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Getting image source signatures 2023-07-20T18:50:52.077 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Copying blob sha256:9a3af2d1f966fade87cf08b715899d7eff2dde16dabc6e2e4c3ec2059da8adc1 2023-07-20T18:50:52.077 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-20T18:50:52.078 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Copying config sha256:3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T18:50:52.078 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-20T18:50:52.079 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Storing signatures 2023-07-20T18:50:53.154 INFO:teuthology.orchestra.run.smithi057.stdout:ceph: stdout ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T18:50:53.154 INFO:teuthology.orchestra.run.smithi057.stdout:Ceph version: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T18:50:53.154 INFO:teuthology.orchestra.run.smithi057.stdout:Extracting ceph user uid/gid from container image... 2023-07-20T18:50:53.933 INFO:teuthology.orchestra.run.smithi057.stdout:stat: stdout 167 167 2023-07-20T18:50:53.933 INFO:teuthology.orchestra.run.smithi057.stdout:Creating initial keys... 2023-07-20T18:50:54.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph-authtool: stdout AQAOgrlksKFiEhAApxePeuz09FBMB/hcQotN6g== 2023-07-20T18:50:55.580 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph-authtool: stdout AQAPgrlkbshvChAAgm+bEPMiCD3hd2m+kjlMsQ== 2023-07-20T18:50:56.458 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph-authtool: stdout AQAPgrlkgpcwOBAAV3IpW6KpQBG1IbdIJzgNSA== 2023-07-20T18:50:56.459 INFO:teuthology.orchestra.run.smithi057.stdout:Creating initial monmap... 2023-07-20T18:50:57.333 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T18:50:57.334 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-20T18:50:57.334 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:50:57.334 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T18:50:57.334 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool for a [v2:172.21.15.57:3300,v1:172.21.15.57:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T18:50:57.334 INFO:teuthology.orchestra.run.smithi057.stdout:setting min_mon_release = pacific 2023-07-20T18:50:57.335 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: set fsid to 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:50:57.335 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T18:50:57.335 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:50:57.335 INFO:teuthology.orchestra.run.smithi057.stdout:Creating mon... 2023-07-20T18:50:59.121 INFO:teuthology.orchestra.run.smithi057.stdout:create mon.a on 2023-07-20T18:50:59.380 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-20T18:50:59.586 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-20T18:50:59.837 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-38a58c3c-272e-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-38a58c3c-272e-11ee-9b34-001a4aab830c.target. 2023-07-20T18:50:59.838 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-38a58c3c-272e-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-38a58c3c-272e-11ee-9b34-001a4aab830c.target. 2023-07-20T18:51:00.237 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.a 2023-07-20T18:51:00.237 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Failed to reset failed state of unit ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.a.service: Unit ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.a.service not loaded. 2023-07-20T18:51:00.449 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-38a58c3c-272e-11ee-9b34-001a4aab830c.target.wants/ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@.service. 2023-07-20T18:51:00.721 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:00 smithi057 systemd[1]: Starting Ceph mon.a for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:51:01.238 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:00 smithi057 podman[125796]: 2023-07-20T18:51:01.366 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T18:51:01.366 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout disabled 2023-07-20T18:51:01.382 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T18:51:01.383 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2023-07-20T18:51:01.383 INFO:teuthology.orchestra.run.smithi057.stdout:firewalld.service is not enabled 2023-07-20T18:51:01.383 INFO:teuthology.orchestra.run.smithi057.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T18:51:01.383 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mon to start... 2023-07-20T18:51:01.384 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mon... 2023-07-20T18:51:01.637 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:01 smithi057 bash[125796]: fef249637734a7b5ea18843d74776e53f3158345400837bb48048d599417ea1e 2023-07-20T18:51:01.638 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:01 smithi057 ceph-mon[125836]: mkfs 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:51:01.638 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:01 smithi057 ceph-mon[125836]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T18:51:01.638 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:01 smithi057 systemd[1]: Started Ceph mon.a for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:51:02.469 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout cluster: 2023-07-20T18:51:02.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout id: 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:51:02.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-20T18:51:02.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:02.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout services: 2023-07-20T18:51:02.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.7265s) 2023-07-20T18:51:02.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-20T18:51:02.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-20T18:51:02.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:02.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout data: 2023-07-20T18:51:02.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-20T18:51:02.472 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-20T18:51:02.472 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-20T18:51:02.472 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout pgs: 2023-07-20T18:51:02.472 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:02.472 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.881+0000 7ff9a6a86700 1 Processor -- start 2023-07-20T18:51:02.472 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.882+0000 7ff9a6a86700 1 -- start start 2023-07-20T18:51:02.473 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.882+0000 7ff9a6a86700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a007a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:02.473 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.882+0000 7ff9a6a86700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a007a8d0 con 0x7ff9a007bea0 2023-07-20T18:51:02.473 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.883+0000 7ff9a4822700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a007a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:02.473 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.883+0000 7ff9a4822700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a007a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57016/0 (socket says 172.21.15.57:57016) 2023-07-20T18:51:02.473 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.883+0000 7ff9a4822700 1 -- 172.21.15.57:0/1038417156 learned_addr learned my addr 172.21.15.57:0/1038417156 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:02.473 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.884+0000 7ff9a4822700 1 -- 172.21.15.57:0/1038417156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a007aa10 con 0x7ff9a007bea0 2023-07-20T18:51:02.473 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.884+0000 7ff9a4822700 1 --2- 172.21.15.57:0/1038417156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a007a300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff9940098a0 tx=0x7ff994009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d631d733eb8b3d3b server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:02.473 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.884+0000 7ff99f7fe700 1 -- 172.21.15.57:0/1038417156 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff994004160 con 0x7ff9a007bea0 2023-07-20T18:51:02.474 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.884+0000 7ff99f7fe700 1 -- 172.21.15.57:0/1038417156 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940042c0 con 0x7ff9a007bea0 2023-07-20T18:51:02.474 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.884+0000 7ff99f7fe700 1 -- 172.21.15.57:0/1038417156 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9940045a0 con 0x7ff9a007bea0 2023-07-20T18:51:02.474 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.885+0000 7ff9a6a86700 1 -- 172.21.15.57:0/1038417156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 msgr2=0x7ff9a007a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:02.474 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.885+0000 7ff9a6a86700 1 --2- 172.21.15.57:0/1038417156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a007a300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff9940098a0 tx=0x7ff994009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:02.474 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.885+0000 7ff9a6a86700 1 -- 172.21.15.57:0/1038417156 shutdown_connections 2023-07-20T18:51:02.474 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.885+0000 7ff9a6a86700 1 --2- 172.21.15.57:0/1038417156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a007a300 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:02.474 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.885+0000 7ff9a6a86700 1 -- 172.21.15.57:0/1038417156 >> 172.21.15.57:0/1038417156 conn(0x7ff9a0101610 msgr2=0x7ff9a0103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:02.475 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.885+0000 7ff9a6a86700 1 -- 172.21.15.57:0/1038417156 shutdown_connections 2023-07-20T18:51:02.475 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.885+0000 7ff9a6a86700 1 -- 172.21.15.57:0/1038417156 wait complete. 2023-07-20T18:51:02.475 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.886+0000 7ff9a6a86700 1 Processor -- start 2023-07-20T18:51:02.475 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.886+0000 7ff9a6a86700 1 -- start start 2023-07-20T18:51:02.475 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.886+0000 7ff9a6a86700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a019b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:02.476 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.886+0000 7ff9a6a86700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a019bbc0 con 0x7ff9a007bea0 2023-07-20T18:51:02.476 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.887+0000 7ff9a4822700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a019b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:02.476 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.887+0000 7ff9a4822700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a019b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57018/0 (socket says 172.21.15.57:57018) 2023-07-20T18:51:02.476 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.887+0000 7ff9a4822700 1 -- 172.21.15.57:0/3626276175 learned_addr learned my addr 172.21.15.57:0/3626276175 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:02.476 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.887+0000 7ff9a4822700 1 -- 172.21.15.57:0/3626276175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff994009550 con 0x7ff9a007bea0 2023-07-20T18:51:02.477 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.887+0000 7ff9a4822700 1 --2- 172.21.15.57:0/3626276175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a019b680 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff994004000 tx=0x7ff994004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:02.477 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.887+0000 7ff99dffb700 1 -- 172.21.15.57:0/3626276175 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff994011630 con 0x7ff9a007bea0 2023-07-20T18:51:02.477 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.888+0000 7ff99dffb700 1 -- 172.21.15.57:0/3626276175 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994011790 con 0x7ff9a007bea0 2023-07-20T18:51:02.477 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.888+0000 7ff99dffb700 1 -- 172.21.15.57:0/3626276175 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff994011a30 con 0x7ff9a007bea0 2023-07-20T18:51:02.477 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.888+0000 7ff9a6a86700 1 -- 172.21.15.57:0/3626276175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a019bdc0 con 0x7ff9a007bea0 2023-07-20T18:51:02.477 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.888+0000 7ff9a6a86700 1 -- 172.21.15.57:0/3626276175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a019c1e0 con 0x7ff9a007bea0 2023-07-20T18:51:02.477 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.888+0000 7ff99dffb700 1 -- 172.21.15.57:0/3626276175 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff994022070 con 0x7ff9a007bea0 2023-07-20T18:51:02.477 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.888+0000 7ff99dffb700 1 -- 172.21.15.57:0/3626276175 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff99401c450 con 0x7ff9a007bea0 2023-07-20T18:51:02.478 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.889+0000 7ff9a6a86700 1 -- 172.21.15.57:0/3626276175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a01955a0 con 0x7ff9a007bea0 2023-07-20T18:51:02.478 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.892+0000 7ff99dffb700 1 -- 172.21.15.57:0/3626276175 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7ff99401b450 con 0x7ff9a007bea0 2023-07-20T18:51:02.478 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.968+0000 7ff9a6a86700 1 -- 172.21.15.57:0/3626276175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7ff9a002cb60 con 0x7ff9a007bea0 2023-07-20T18:51:02.478 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.968+0000 7ff99dffb700 1 -- 172.21.15.57:0/3626276175 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+315 (secure 0 0 0) 0x7ff99401b5f0 con 0x7ff9a007bea0 2023-07-20T18:51:02.478 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.969+0000 7ff9a6a86700 1 -- 172.21.15.57:0/3626276175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 msgr2=0x7ff9a019b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:02.478 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.969+0000 7ff9a6a86700 1 --2- 172.21.15.57:0/3626276175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a019b680 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff994004000 tx=0x7ff994004750 comp rx=0 tx=0).stop 2023-07-20T18:51:02.478 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.969+0000 7ff9a6a86700 1 -- 172.21.15.57:0/3626276175 shutdown_connections 2023-07-20T18:51:02.479 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.969+0000 7ff9a6a86700 1 --2- 172.21.15.57:0/3626276175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9a007bea0 0x7ff9a019b680 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:02.479 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.969+0000 7ff9a6a86700 1 -- 172.21.15.57:0/3626276175 >> 172.21.15.57:0/3626276175 conn(0x7ff9a0101610 msgr2=0x7ff9a0102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:02.479 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.969+0000 7ff9a6a86700 1 -- 172.21.15.57:0/3626276175 shutdown_connections 2023-07-20T18:51:02.479 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:01.969+0000 7ff9a6a86700 1 -- 172.21.15.57:0/3626276175 wait complete. 2023-07-20T18:51:02.479 INFO:teuthology.orchestra.run.smithi057.stdout:mon is available 2023-07-20T18:51:02.479 INFO:teuthology.orchestra.run.smithi057.stdout:Assimilating anything we can from ceph.conf... 2023-07-20T18:51:02.721 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:02 smithi057 ceph-mon[125836]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T18:51:02.721 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:02 smithi057 ceph-mon[125836]: monmap e1: 1 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0]} removed_ranks: {} 2023-07-20T18:51:02.722 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:02 smithi057 ceph-mon[125836]: fsmap 2023-07-20T18:51:02.722 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:02 smithi057 ceph-mon[125836]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T18:51:02.722 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:02 smithi057 ceph-mon[125836]: mgrmap e1: no daemons active 2023-07-20T18:51:02.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:02 smithi057 ceph-mon[125836]: from='client.? 172.21.15.57:0/3626276175' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-20T18:51:03.471 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:03 smithi057 ceph-mon[125836]: from='client.? 172.21.15.57:0/3045628552' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T18:51:03.471 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:03 smithi057 ceph-mon[125836]: from='client.? 172.21.15.57:0/3045628552' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-20T18:51:03.547 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:03.549 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T18:51:03.549 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout fsid = 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:51:03.549 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.57:3300,v1:172.21.15.57:6789] 2023-07-20T18:51:03.549 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T18:51:03.549 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T18:51:03.550 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T18:51:03.550 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T18:51:03.550 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:03.550 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T18:51:03.550 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T18:51:03.550 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:03.550 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T18:51:03.550 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T18:51:03.551 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T18:51:03.551 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T18:51:03.551 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T18:51:03.551 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T18:51:03.551 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.989+0000 7f719d18d700 1 Processor -- start 2023-07-20T18:51:03.551 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.990+0000 7f719d18d700 1 -- start start 2023-07-20T18:51:03.552 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.990+0000 7f719d18d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719807a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:03.552 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.990+0000 7f719d18d700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719807a8b0 con 0x7f719807be80 2023-07-20T18:51:03.556 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.990+0000 7f7196d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719807a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:03.556 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.990+0000 7f7196d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719807a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57024/0 (socket says 172.21.15.57:57024) 2023-07-20T18:51:03.556 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.990+0000 7f7196d9d700 1 -- 172.21.15.57:0/3912338308 learned_addr learned my addr 172.21.15.57:0/3912338308 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:03.557 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.991+0000 7f7196d9d700 1 -- 172.21.15.57:0/3912338308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719807a9f0 con 0x7f719807be80 2023-07-20T18:51:03.557 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.991+0000 7f7196d9d700 1 --2- 172.21.15.57:0/3912338308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719807a2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f71880098a0 tx=0x7f7188009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e455893b45f69867 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:03.557 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.991+0000 7f7195d9b700 1 -- 172.21.15.57:0/3912338308 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7188004160 con 0x7f719807be80 2023-07-20T18:51:03.557 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.991+0000 7f7195d9b700 1 -- 172.21.15.57:0/3912338308 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71880042c0 con 0x7f719807be80 2023-07-20T18:51:03.557 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.991+0000 7f7195d9b700 1 -- 172.21.15.57:0/3912338308 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71880045a0 con 0x7f719807be80 2023-07-20T18:51:03.557 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.992+0000 7f719d18d700 1 -- 172.21.15.57:0/3912338308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 msgr2=0x7f719807a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:03.558 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.992+0000 7f719d18d700 1 --2- 172.21.15.57:0/3912338308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719807a2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f71880098a0 tx=0x7f7188009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:03.558 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.992+0000 7f719d18d700 1 -- 172.21.15.57:0/3912338308 shutdown_connections 2023-07-20T18:51:03.558 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.992+0000 7f719d18d700 1 --2- 172.21.15.57:0/3912338308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719807a2e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:03.558 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.992+0000 7f719d18d700 1 -- 172.21.15.57:0/3912338308 >> 172.21.15.57:0/3912338308 conn(0x7f7198101610 msgr2=0x7f7198103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:03.558 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.992+0000 7f719d18d700 1 -- 172.21.15.57:0/3912338308 shutdown_connections 2023-07-20T18:51:03.559 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.992+0000 7f719d18d700 1 -- 172.21.15.57:0/3912338308 wait complete. 2023-07-20T18:51:03.559 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.993+0000 7f719d18d700 1 Processor -- start 2023-07-20T18:51:03.559 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.993+0000 7f719d18d700 1 -- start start 2023-07-20T18:51:03.559 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.993+0000 7f719d18d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719819b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:03.559 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.993+0000 7f719d18d700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719819bb80 con 0x7f719807be80 2023-07-20T18:51:03.559 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.994+0000 7f7196d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719819b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:03.560 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.994+0000 7f7196d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719819b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57030/0 (socket says 172.21.15.57:57030) 2023-07-20T18:51:03.560 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.994+0000 7f7196d9d700 1 -- 172.21.15.57:0/3045628552 learned_addr learned my addr 172.21.15.57:0/3045628552 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:03.560 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.994+0000 7f7196d9d700 1 -- 172.21.15.57:0/3045628552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7188009550 con 0x7f719807be80 2023-07-20T18:51:03.560 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.994+0000 7f7196d9d700 1 --2- 172.21.15.57:0/3045628552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719819b640 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f719807b350 tx=0x7f7188004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:03.560 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.994+0000 7f717ffff700 1 -- 172.21.15.57:0/3045628552 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7188011630 con 0x7f719807be80 2023-07-20T18:51:03.560 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.994+0000 7f717ffff700 1 -- 172.21.15.57:0/3045628552 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7188011790 con 0x7f719807be80 2023-07-20T18:51:03.560 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.994+0000 7f719d18d700 1 -- 172.21.15.57:0/3045628552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719819bd80 con 0x7f719807be80 2023-07-20T18:51:03.561 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.994+0000 7f717ffff700 1 -- 172.21.15.57:0/3045628552 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7188011a30 con 0x7f719807be80 2023-07-20T18:51:03.561 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.995+0000 7f719d18d700 1 -- 172.21.15.57:0/3045628552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719819c1a0 con 0x7f719807be80 2023-07-20T18:51:03.561 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.995+0000 7f717ffff700 1 -- 172.21.15.57:0/3045628552 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7188022070 con 0x7f719807be80 2023-07-20T18:51:03.561 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.995+0000 7f717ffff700 1 -- 172.21.15.57:0/3045628552 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f718801c450 con 0x7f719807be80 2023-07-20T18:51:03.561 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.996+0000 7f719d18d700 1 -- 172.21.15.57:0/3045628552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7198195650 con 0x7f719807be80 2023-07-20T18:51:03.561 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:02.998+0000 7f717ffff700 1 -- 172.21.15.57:0/3045628552 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f718801b450 con 0x7f719807be80 2023-07-20T18:51:03.561 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:03.072+0000 7f719d18d700 1 -- 172.21.15.57:0/3045628552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f719802cb00 con 0x7f719807be80 2023-07-20T18:51:03.562 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:03.076+0000 7f717ffff700 1 -- 172.21.15.57:0/3045628552 <== mon.0 v2:172.21.15.57:3300/0 7 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f718801b5f0 con 0x7f719807be80 2023-07-20T18:51:03.562 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:03.076+0000 7f717ffff700 1 -- 172.21.15.57:0/3045628552 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f71880376a0 con 0x7f719807be80 2023-07-20T18:51:03.562 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:03.077+0000 7f719d18d700 1 -- 172.21.15.57:0/3045628552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 msgr2=0x7f719819b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:03.562 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:03.077+0000 7f719d18d700 1 --2- 172.21.15.57:0/3045628552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719819b640 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f719807b350 tx=0x7f7188004750 comp rx=0 tx=0).stop 2023-07-20T18:51:03.562 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:03.078+0000 7f719d18d700 1 -- 172.21.15.57:0/3045628552 shutdown_connections 2023-07-20T18:51:03.562 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:03.078+0000 7f719d18d700 1 --2- 172.21.15.57:0/3045628552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f719807be80 0x7f719819b640 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:03.562 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:03.078+0000 7f719d18d700 1 -- 172.21.15.57:0/3045628552 >> 172.21.15.57:0/3045628552 conn(0x7f7198101610 msgr2=0x7f7198102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:03.563 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:03.078+0000 7f719d18d700 1 -- 172.21.15.57:0/3045628552 shutdown_connections 2023-07-20T18:51:03.563 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:03.078+0000 7f719d18d700 1 -- 172.21.15.57:0/3045628552 wait complete. 2023-07-20T18:51:03.563 INFO:teuthology.orchestra.run.smithi057.stdout:Generating new minimal ceph.conf... 2023-07-20T18:51:05.721 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:05 smithi057 ceph-mon[125836]: from='client.? 172.21.15.57:0/3802189620' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:51:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.123+0000 7fd366392700 1 Processor -- start 2023-07-20T18:51:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.123+0000 7fd366392700 1 -- start start 2023-07-20T18:51:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.123+0000 7fd366392700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36007a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.123+0000 7fd366392700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36007a8d0 con 0x7fd36007bea0 2023-07-20T18:51:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.124+0000 7fd35ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36007a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.124+0000 7fd35ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36007a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53376/0 (socket says 172.21.15.57:53376) 2023-07-20T18:51:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.124+0000 7fd35ffff700 1 -- 172.21.15.57:0/4058943812 learned_addr learned my addr 172.21.15.57:0/4058943812 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.124+0000 7fd35ffff700 1 -- 172.21.15.57:0/4058943812 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36007aa10 con 0x7fd36007bea0 2023-07-20T18:51:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.124+0000 7fd35ffff700 1 --2- 172.21.15.57:0/4058943812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36007a300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd3500098a0 tx=0x7fd350009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bee4382ff6e3a1 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.124+0000 7fd35effd700 1 -- 172.21.15.57:0/4058943812 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd350004160 con 0x7fd36007bea0 2023-07-20T18:51:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.125+0000 7fd35effd700 1 -- 172.21.15.57:0/4058943812 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fd3500042c0 con 0x7fd36007bea0 2023-07-20T18:51:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.125+0000 7fd35effd700 1 -- 172.21.15.57:0/4058943812 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd350011ac0 con 0x7fd36007bea0 2023-07-20T18:51:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.125+0000 7fd366392700 1 -- 172.21.15.57:0/4058943812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 msgr2=0x7fd36007a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.125+0000 7fd366392700 1 --2- 172.21.15.57:0/4058943812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36007a300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd3500098a0 tx=0x7fd350009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.125+0000 7fd366392700 1 -- 172.21.15.57:0/4058943812 shutdown_connections 2023-07-20T18:51:05.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.125+0000 7fd366392700 1 --2- 172.21.15.57:0/4058943812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36007a300 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:05.743 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.125+0000 7fd366392700 1 -- 172.21.15.57:0/4058943812 >> 172.21.15.57:0/4058943812 conn(0x7fd360101610 msgr2=0x7fd360103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:05.743 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.125+0000 7fd366392700 1 -- 172.21.15.57:0/4058943812 shutdown_connections 2023-07-20T18:51:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.125+0000 7fd366392700 1 -- 172.21.15.57:0/4058943812 wait complete. 2023-07-20T18:51:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.126+0000 7fd366392700 1 Processor -- start 2023-07-20T18:51:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.127+0000 7fd366392700 1 -- start start 2023-07-20T18:51:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.127+0000 7fd366392700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36019bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.127+0000 7fd366392700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36019c470 con 0x7fd36007bea0 2023-07-20T18:51:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.127+0000 7fd35ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36019bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.127+0000 7fd35ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36019bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53380/0 (socket says 172.21.15.57:53380) 2023-07-20T18:51:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.127+0000 7fd35ffff700 1 -- 172.21.15.57:0/3802189620 learned_addr learned my addr 172.21.15.57:0/3802189620 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.127+0000 7fd35ffff700 1 -- 172.21.15.57:0/3802189620 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd350009550 con 0x7fd36007bea0 2023-07-20T18:51:05.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.128+0000 7fd35ffff700 1 --2- 172.21.15.57:0/3802189620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36019bf30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd350003f70 tx=0x7fd350011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.128+0000 7fd35d7fa700 1 -- 172.21.15.57:0/3802189620 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd3500039f0 con 0x7fd36007bea0 2023-07-20T18:51:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.128+0000 7fd35d7fa700 1 -- 172.21.15.57:0/3802189620 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fd350003b50 con 0x7fd36007bea0 2023-07-20T18:51:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.128+0000 7fd366392700 1 -- 172.21.15.57:0/3802189620 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36019c670 con 0x7fd36007bea0 2023-07-20T18:51:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.128+0000 7fd35d7fa700 1 -- 172.21.15.57:0/3802189620 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd35001a690 con 0x7fd36007bea0 2023-07-20T18:51:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.128+0000 7fd366392700 1 -- 172.21.15.57:0/3802189620 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36019ca90 con 0x7fd36007bea0 2023-07-20T18:51:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.128+0000 7fd35d7fa700 1 -- 172.21.15.57:0/3802189620 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd35001ac00 con 0x7fd36007bea0 2023-07-20T18:51:05.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.129+0000 7fd35d7fa700 1 -- 172.21.15.57:0/3802189620 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd350019700 con 0x7fd36007bea0 2023-07-20T18:51:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.129+0000 7fd366392700 1 -- 172.21.15.57:0/3802189620 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd360061da0 con 0x7fd36007bea0 2023-07-20T18:51:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.132+0000 7fd35d7fa700 1 -- 172.21.15.57:0/3802189620 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd3500249f0 con 0x7fd36007bea0 2023-07-20T18:51:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.205+0000 7fd366392700 1 -- 172.21.15.57:0/3802189620 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd36019f5c0 con 0x7fd36007bea0 2023-07-20T18:51:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.205+0000 7fd35d7fa700 1 -- 172.21.15.57:0/3802189620 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fd350024430 con 0x7fd36007bea0 2023-07-20T18:51:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.206+0000 7fd366392700 1 -- 172.21.15.57:0/3802189620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 msgr2=0x7fd36019bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:05.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.206+0000 7fd366392700 1 --2- 172.21.15.57:0/3802189620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36019bf30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd350003f70 tx=0x7fd350011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.206+0000 7fd366392700 1 -- 172.21.15.57:0/3802189620 shutdown_connections 2023-07-20T18:51:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.206+0000 7fd366392700 1 --2- 172.21.15.57:0/3802189620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd36007bea0 0x7fd36019bf30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.206+0000 7fd366392700 1 -- 172.21.15.57:0/3802189620 >> 172.21.15.57:0/3802189620 conn(0x7fd360101610 msgr2=0x7fd360102400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.207+0000 7fd366392700 1 -- 172.21.15.57:0/3802189620 shutdown_connections 2023-07-20T18:51:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:05.207+0000 7fd366392700 1 -- 172.21.15.57:0/3802189620 wait complete. 2023-07-20T18:51:05.748 INFO:teuthology.orchestra.run.smithi057.stdout:Restarting the monitor... 2023-07-20T18:51:06.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:05 smithi057 systemd[1]: Stopping Ceph mon.a for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:51:06.471 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:06 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-a[125812]: 2023-07-20T18:51:06.172+0000 7f54b90c0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:51:06.471 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:06 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-a[125812]: 2023-07-20T18:51:06.172+0000 7f54b90c0700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-20T18:51:06.753 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:06 smithi057 bash[126174]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-a 2023-07-20T18:51:07.471 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 systemd[1]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.a.service: Succeeded. 2023-07-20T18:51:07.472 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 systemd[1]: Stopped Ceph mon.a for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:51:07.472 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 systemd[1]: Starting Ceph mon.a for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:51:07.757 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 podman[126300]: 2023-07-20T18:51:07.757 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 bash[126300]: 1beb97f2d0ea7185ef822f37553bb56929ad194ce8c93d850a77a11b69b3b0ab 2023-07-20T18:51:07.757 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T18:51:07.768 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T18:51:07.768 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: pidfile_write: ignore empty --pid-file 2023-07-20T18:51:07.768 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: load: jerasure load: lrc 2023-07-20T18:51:07.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: RocksDB version: 7.9.2 2023-07-20T18:51:07.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Git sha 0 2023-07-20T18:51:07.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T18:51:07.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: DB SUMMARY 2023-07-20T18:51:07.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: DB Session ID: M6DNT17HTO6DHIECTOSB 2023-07-20T18:51:07.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: CURRENT file: CURRENT 2023-07-20T18:51:07.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T18:51:07.824 INFO:teuthology.orchestra.run.smithi057.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-20T18:51:08.071 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-20T18:51:08.071 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-20T18:51:08.071 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98809 ; 2023-07-20T18:51:08.071 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.error_if_exists: 0 2023-07-20T18:51:08.072 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.create_if_missing: 0 2023-07-20T18:51:08.072 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.paranoid_checks: 1 2023-07-20T18:51:08.072 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T18:51:08.072 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T18:51:08.072 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T18:51:08.072 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.env: 0x5602384c0c60 2023-07-20T18:51:08.073 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T18:51:08.073 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.info_log: 0x5602390db2c0 2023-07-20T18:51:08.073 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T18:51:08.073 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.statistics: (nil) 2023-07-20T18:51:08.073 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.use_fsync: 0 2023-07-20T18:51:08.073 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_log_file_size: 0 2023-07-20T18:51:08.073 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T18:51:08.074 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T18:51:08.074 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T18:51:08.074 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T18:51:08.074 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.allow_fallocate: 1 2023-07-20T18:51:08.074 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T18:51:08.074 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T18:51:08.074 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.use_direct_reads: 0 2023-07-20T18:51:08.074 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T18:51:08.075 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T18:51:08.075 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.db_log_dir: 2023-07-20T18:51:08.075 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.wal_dir: 2023-07-20T18:51:08.075 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T18:51:08.075 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T18:51:08.075 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T18:51:08.075 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T18:51:08.075 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T18:51:08.076 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T18:51:08.076 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T18:51:08.076 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T18:51:08.076 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.write_buffer_manager: 0x560238ffc3c0 2023-07-20T18:51:08.076 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T18:51:08.076 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T18:51:08.076 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T18:51:08.077 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T18:51:08.077 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T18:51:08.077 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T18:51:08.077 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T18:51:08.077 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T18:51:08.077 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.unordered_write: 0 2023-07-20T18:51:08.077 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T18:51:08.077 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T18:51:08.078 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T18:51:08.078 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T18:51:08.078 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.row_cache: None 2023-07-20T18:51:08.078 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.wal_filter: None 2023-07-20T18:51:08.078 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T18:51:08.078 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T18:51:08.078 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.two_write_queues: 0 2023-07-20T18:51:08.078 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T18:51:08.079 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.wal_compression: 0 2023-07-20T18:51:08.079 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.atomic_flush: 0 2023-07-20T18:51:08.079 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T18:51:08.079 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T18:51:08.079 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T18:51:08.079 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.log_readahead_size: 0 2023-07-20T18:51:08.079 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T18:51:08.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T18:51:08.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T18:51:08.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T18:51:08.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T18:51:08.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T18:51:08.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T18:51:08.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_background_jobs: 2 2023-07-20T18:51:08.081 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_background_compactions: -1 2023-07-20T18:51:08.081 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_subcompactions: 1 2023-07-20T18:51:08.081 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T18:51:08.081 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T18:51:08.081 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T18:51:08.081 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T18:51:08.081 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T18:51:08.081 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T18:51:08.082 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T18:51:08.082 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T18:51:08.082 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_open_files: -1 2023-07-20T18:51:08.082 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T18:51:08.082 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T18:51:08.082 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T18:51:08.082 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T18:51:08.082 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_background_flushes: -1 2023-07-20T18:51:08.083 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Compression algorithms supported: 2023-07-20T18:51:08.083 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T18:51:08.083 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: kZSTD supported: 0 2023-07-20T18:51:08.083 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: kXpressCompression supported: 0 2023-07-20T18:51:08.083 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T18:51:08.083 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: kZlibCompression supported: 1 2023-07-20T18:51:08.083 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: kSnappyCompression supported: 1 2023-07-20T18:51:08.084 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: kLZ4Compression supported: 1 2023-07-20T18:51:08.084 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: kBZip2Compression supported: 0 2023-07-20T18:51:08.084 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T18:51:08.084 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T18:51:08.084 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-20T18:51:08.084 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T18:51:08.084 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T18:51:08.084 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.merge_operator: 2023-07-20T18:51:08.085 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_filter: None 2023-07-20T18:51:08.085 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T18:51:08.085 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T18:51:08.085 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T18:51:08.085 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T18:51:08.085 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5602390db3c0) 2023-07-20T18:51:08.085 INFO:journalctl@ceph.mon.a.smithi057.stdout: cache_index_and_filter_blocks: 1 2023-07-20T18:51:08.086 INFO:journalctl@ceph.mon.a.smithi057.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T18:51:08.086 INFO:journalctl@ceph.mon.a.smithi057.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T18:51:08.086 INFO:journalctl@ceph.mon.a.smithi057.stdout: pin_top_level_index_and_filter: 1 2023-07-20T18:51:08.086 INFO:journalctl@ceph.mon.a.smithi057.stdout: index_type: 0 2023-07-20T18:51:08.086 INFO:journalctl@ceph.mon.a.smithi057.stdout: data_block_index_type: 0 2023-07-20T18:51:08.086 INFO:journalctl@ceph.mon.a.smithi057.stdout: index_shortening: 1 2023-07-20T18:51:08.086 INFO:journalctl@ceph.mon.a.smithi057.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T18:51:08.086 INFO:journalctl@ceph.mon.a.smithi057.stdout: checksum: 4 2023-07-20T18:51:08.087 INFO:journalctl@ceph.mon.a.smithi057.stdout: no_block_cache: 0 2023-07-20T18:51:08.087 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_cache: 0x56023909b350 2023-07-20T18:51:08.087 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_cache_name: BinnedLRUCache 2023-07-20T18:51:08.087 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_cache_options: 2023-07-20T18:51:08.087 INFO:journalctl@ceph.mon.a.smithi057.stdout: capacity : 536870912 2023-07-20T18:51:08.087 INFO:journalctl@ceph.mon.a.smithi057.stdout: num_shard_bits : 4 2023-07-20T18:51:08.088 INFO:journalctl@ceph.mon.a.smithi057.stdout: strict_capacity_limit : 0 2023-07-20T18:51:08.088 INFO:journalctl@ceph.mon.a.smithi057.stdout: high_pri_pool_ratio: 0.000 2023-07-20T18:51:08.088 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_cache_compressed: (nil) 2023-07-20T18:51:08.088 INFO:journalctl@ceph.mon.a.smithi057.stdout: persistent_cache: (nil) 2023-07-20T18:51:08.088 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_size: 4096 2023-07-20T18:51:08.088 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_size_deviation: 10 2023-07-20T18:51:08.088 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_restart_interval: 16 2023-07-20T18:51:08.089 INFO:journalctl@ceph.mon.a.smithi057.stdout: index_block_restart_interval: 1 2023-07-20T18:51:08.089 INFO:journalctl@ceph.mon.a.smithi057.stdout: metadata_block_size: 4096 2023-07-20T18:51:08.089 INFO:journalctl@ceph.mon.a.smithi057.stdout: partition_filters: 0 2023-07-20T18:51:08.089 INFO:journalctl@ceph.mon.a.smithi057.stdout: use_delta_encoding: 1 2023-07-20T18:51:08.089 INFO:journalctl@ceph.mon.a.smithi057.stdout: filter_policy: bloomfilter 2023-07-20T18:51:08.089 INFO:journalctl@ceph.mon.a.smithi057.stdout: whole_key_filtering: 1 2023-07-20T18:51:08.089 INFO:journalctl@ceph.mon.a.smithi057.stdout: verify_compression: 0 2023-07-20T18:51:08.089 INFO:journalctl@ceph.mon.a.smithi057.stdout: read_amp_bytes_per_bit: 0 2023-07-20T18:51:08.090 INFO:journalctl@ceph.mon.a.smithi057.stdout: format_version: 5 2023-07-20T18:51:08.090 INFO:journalctl@ceph.mon.a.smithi057.stdout: enable_index_compression: 1 2023-07-20T18:51:08.090 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_align: 0 2023-07-20T18:51:08.090 INFO:journalctl@ceph.mon.a.smithi057.stdout: max_auto_readahead_size: 262144 2023-07-20T18:51:08.090 INFO:journalctl@ceph.mon.a.smithi057.stdout: prepopulate_block_cache: 0 2023-07-20T18:51:08.090 INFO:journalctl@ceph.mon.a.smithi057.stdout: initial_auto_readahead_size: 8192 2023-07-20T18:51:08.090 INFO:journalctl@ceph.mon.a.smithi057.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T18:51:08.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T18:51:08.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T18:51:08.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compression: NoCompression 2023-07-20T18:51:08.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T18:51:08.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T18:51:08.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T18:51:08.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.num_levels: 7 2023-07-20T18:51:08.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T18:51:08.092 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T18:51:08.092 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T18:51:08.092 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T18:51:08.092 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T18:51:08.092 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T18:51:08.092 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T18:51:08.092 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T18:51:08.092 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T18:51:08.093 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T18:51:08.093 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T18:51:08.093 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T18:51:08.093 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T18:51:08.093 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T18:51:08.093 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T18:51:08.093 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T18:51:08.093 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T18:51:08.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T18:51:08.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T18:51:08.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T18:51:08.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T18:51:08.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T18:51:08.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T18:51:08.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T18:51:08.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T18:51:08.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T18:51:08.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T18:51:08.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T18:51:08.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T18:51:08.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T18:51:08.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T18:51:08.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T18:51:08.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T18:51:08.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T18:51:08.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T18:51:08.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T18:51:08.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T18:51:08.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T18:51:08.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T18:51:08.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T18:51:08.097 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T18:51:08.097 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T18:51:08.097 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T18:51:08.097 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T18:51:08.097 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T18:51:08.097 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T18:51:08.097 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T18:51:08.098 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T18:51:08.098 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T18:51:08.098 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T18:51:08.098 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T18:51:08.098 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T18:51:08.098 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T18:51:08.098 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.table_properties_collectors: 2023-07-20T18:51:08.098 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.inplace_update_support: 0 2023-07-20T18:51:08.099 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T18:51:08.099 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T18:51:08.099 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T18:51:08.099 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T18:51:08.099 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.bloom_locality: 0 2023-07-20T18:51:08.099 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.max_successive_merges: 0 2023-07-20T18:51:08.099 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T18:51:08.099 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T18:51:08.100 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T18:51:08.100 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T18:51:08.100 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.ttl: 2592000 2023-07-20T18:51:08.100 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T18:51:08.100 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T18:51:08.100 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T18:51:08.100 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.enable_blob_files: false 2023-07-20T18:51:08.101 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.min_blob_size: 0 2023-07-20T18:51:08.101 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T18:51:08.101 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T18:51:08.101 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T18:51:08.101 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T18:51:08.101 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T18:51:08.101 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T18:51:08.101 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T18:51:08.102 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T18:51:08.102 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-20T18:51:08.102 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-20T18:51:08.102 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e122101f-57da-449b-b3d6-89b5d7e3e3ad 2023-07-20T18:51:08.102 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879067730600, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-20T18:51:08.102 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-20T18:51:08.102 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879067732391, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94004, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92015, "index_size": 271, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13656, "raw_average_key_size": 50, "raw_value_size": 84604, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689879067, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e122101f-57da-449b-b3d6-89b5d7e3e3ad", "db_session_id": "M6DNT17HTO6DHIECTOSB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-20T18:51:08.103 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879067732582, "job": 1, "event": "recovery_finished"} 2023-07-20T18:51:08.103 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-20T18:51:08.103 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T18:51:08.103 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560239134000 2023-07-20T18:51:08.103 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: DB pointer 0x560239120000 2023-07-20T18:51:08.103 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T18:51:08.103 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T18:51:08.103 INFO:journalctl@ceph.mon.a.smithi057.stdout: ** DB Stats ** 2023-07-20T18:51:08.104 INFO:journalctl@ceph.mon.a.smithi057.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T18:51:08.104 INFO:journalctl@ceph.mon.a.smithi057.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T18:51:08.104 INFO:journalctl@ceph.mon.a.smithi057.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T18:51:08.104 INFO:journalctl@ceph.mon.a.smithi057.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T18:51:08.104 INFO:journalctl@ceph.mon.a.smithi057.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T18:51:08.104 INFO:journalctl@ceph.mon.a.smithi057.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T18:51:08.104 INFO:journalctl@ceph.mon.a.smithi057.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T18:51:08.105 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2023-07-20T18:51:08.105 INFO:journalctl@ceph.mon.a.smithi057.stdout: ** Compaction Stats [default] ** 2023-07-20T18:51:08.105 INFO:journalctl@ceph.mon.a.smithi057.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T18:51:08.105 INFO:journalctl@ceph.mon.a.smithi057.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T18:51:08.105 INFO:journalctl@ceph.mon.a.smithi057.stdout: L0 2/0 93.64 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 74.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T18:51:08.105 INFO:journalctl@ceph.mon.a.smithi057.stdout: Sum 2/0 93.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 74.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T18:51:08.105 INFO:journalctl@ceph.mon.a.smithi057.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 74.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T18:51:08.106 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2023-07-20T18:51:08.106 INFO:journalctl@ceph.mon.a.smithi057.stdout: ** Compaction Stats [default] ** 2023-07-20T18:51:08.106 INFO:journalctl@ceph.mon.a.smithi057.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T18:51:08.106 INFO:journalctl@ceph.mon.a.smithi057.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T18:51:08.106 INFO:journalctl@ceph.mon.a.smithi057.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 74.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T18:51:08.106 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2023-07-20T18:51:08.106 INFO:journalctl@ceph.mon.a.smithi057.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T18:51:08.107 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2023-07-20T18:51:08.107 INFO:journalctl@ceph.mon.a.smithi057.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T18:51:08.107 INFO:journalctl@ceph.mon.a.smithi057.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T18:51:08.107 INFO:journalctl@ceph.mon.a.smithi057.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T18:51:08.107 INFO:journalctl@ceph.mon.a.smithi057.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T18:51:08.107 INFO:journalctl@ceph.mon.a.smithi057.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T18:51:08.107 INFO:journalctl@ceph.mon.a.smithi057.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T18:51:08.107 INFO:journalctl@ceph.mon.a.smithi057.stdout: Cumulative compaction: 0.00 GB write, 13.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T18:51:08.108 INFO:journalctl@ceph.mon.a.smithi057.stdout: Interval compaction: 0.00 GB write, 13.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T18:51:08.108 INFO:journalctl@ceph.mon.a.smithi057.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T18:51:08.108 INFO:journalctl@ceph.mon.a.smithi057.stdout: Block cache BinnedLRUCache@0x56023909b350#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2023-07-20T18:51:08.108 INFO:journalctl@ceph.mon.a.smithi057.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-20T18:51:08.108 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2023-07-20T18:51:08.108 INFO:journalctl@ceph.mon.a.smithi057.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T18:51:08.108 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: starting mon.a rank 0 at public addrs [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] at bind addrs [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:51:08.109 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: mon.a@-1(???) e1 preinit fsid 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:51:08.109 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: mon.a@-1(???).mds e1 new map 2023-07-20T18:51:08.109 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: mon.a@-1(???).mds e1 print_map 2023-07-20T18:51:08.109 INFO:journalctl@ceph.mon.a.smithi057.stdout: e1 2023-07-20T18:51:08.109 INFO:journalctl@ceph.mon.a.smithi057.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T18:51:08.109 INFO:journalctl@ceph.mon.a.smithi057.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T18:51:08.109 INFO:journalctl@ceph.mon.a.smithi057.stdout: legacy client fscid: -1 2023-07-20T18:51:08.109 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2023-07-20T18:51:08.110 INFO:journalctl@ceph.mon.a.smithi057.stdout: No filesystems configured 2023-07-20T18:51:08.110 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T18:51:08.110 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:51:08.110 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:51:08.110 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:51:08.110 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-20T18:51:08.110 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 systemd[1]: Started Ceph mon.a for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:51:08.110 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T18:51:08.111 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: monmap e1: 1 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0]} removed_ranks: {} 2023-07-20T18:51:08.111 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: fsmap 2023-07-20T18:51:08.111 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T18:51:08.111 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:07 smithi057 ceph-mon[126340]: mgrmap e1: no daemons active 2023-07-20T18:51:08.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.310+0000 7fe5d0cdc700 1 Processor -- start 2023-07-20T18:51:08.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.310+0000 7fe5d0cdc700 1 -- start start 2023-07-20T18:51:08.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.310+0000 7fe5d0cdc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:08.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.310+0000 7fe5d0cdc700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc07a8b0 con 0x7fe5cc07be80 2023-07-20T18:51:08.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.311+0000 7fe5ca59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:08.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.311+0000 7fe5ca59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53394/0 (socket says 172.21.15.57:53394) 2023-07-20T18:51:08.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.311+0000 7fe5ca59c700 1 -- 172.21.15.57:0/2791282630 learned_addr learned my addr 172.21.15.57:0/2791282630 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:08.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.311+0000 7fe5ca59c700 1 -- 172.21.15.57:0/2791282630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc07a9f0 con 0x7fe5cc07be80 2023-07-20T18:51:08.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.312+0000 7fe5ca59c700 1 --2- 172.21.15.57:0/2791282630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc07a2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0098a0 tx=0x7fe5bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d0e87dbf5e7723 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:08.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.312+0000 7fe5c959a700 1 -- 172.21.15.57:0/2791282630 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5bc004160 con 0x7fe5cc07be80 2023-07-20T18:51:08.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.312+0000 7fe5c959a700 1 -- 172.21.15.57:0/2791282630 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fe5bc0042c0 con 0x7fe5cc07be80 2023-07-20T18:51:08.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.312+0000 7fe5c959a700 1 -- 172.21.15.57:0/2791282630 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5bc011ac0 con 0x7fe5cc07be80 2023-07-20T18:51:08.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.312+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/2791282630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 msgr2=0x7fe5cc07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:08.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.312+0000 7fe5d0cdc700 1 --2- 172.21.15.57:0/2791282630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc07a2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0098a0 tx=0x7fe5bc009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:08.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.313+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/2791282630 shutdown_connections 2023-07-20T18:51:08.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.313+0000 7fe5d0cdc700 1 --2- 172.21.15.57:0/2791282630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc07a2e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:08.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.313+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/2791282630 >> 172.21.15.57:0/2791282630 conn(0x7fe5cc101610 msgr2=0x7fe5cc103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:08.811 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.313+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/2791282630 shutdown_connections 2023-07-20T18:51:08.811 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.313+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/2791282630 wait complete. 2023-07-20T18:51:08.812 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.314+0000 7fe5d0cdc700 1 Processor -- start 2023-07-20T18:51:08.812 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.314+0000 7fe5d0cdc700 1 -- start start 2023-07-20T18:51:08.812 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.314+0000 7fe5d0cdc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc19bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:08.812 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.314+0000 7fe5d0cdc700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc19c400 con 0x7fe5cc07be80 2023-07-20T18:51:08.812 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.314+0000 7fe5ca59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc19bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:08.813 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.314+0000 7fe5ca59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc19bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53396/0 (socket says 172.21.15.57:53396) 2023-07-20T18:51:08.813 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.314+0000 7fe5ca59c700 1 -- 172.21.15.57:0/3531472197 learned_addr learned my addr 172.21.15.57:0/3531472197 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:08.813 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.315+0000 7fe5ca59c700 1 -- 172.21.15.57:0/3531472197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc009550 con 0x7fe5cc07be80 2023-07-20T18:51:08.813 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.315+0000 7fe5ca59c700 1 --2- 172.21.15.57:0/3531472197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc19bec0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009130 tx=0x7fe5bc011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:08.814 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.315+0000 7fe5bb7fe700 1 -- 172.21.15.57:0/3531472197 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5bc003aa0 con 0x7fe5cc07be80 2023-07-20T18:51:08.814 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.315+0000 7fe5bb7fe700 1 -- 172.21.15.57:0/3531472197 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fe5bc003c00 con 0x7fe5cc07be80 2023-07-20T18:51:08.814 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.315+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/3531472197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5cc19c600 con 0x7fe5cc07be80 2023-07-20T18:51:08.814 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.315+0000 7fe5bb7fe700 1 -- 172.21.15.57:0/3531472197 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5bc024800 con 0x7fe5cc07be80 2023-07-20T18:51:08.814 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.315+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/3531472197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5cc19ca20 con 0x7fe5cc07be80 2023-07-20T18:51:08.814 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.316+0000 7fe5bb7fe700 1 -- 172.21.15.57:0/3531472197 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe5bc02d3f0 con 0x7fe5cc07be80 2023-07-20T18:51:08.814 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.316+0000 7fe5bb7fe700 1 -- 172.21.15.57:0/3531472197 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe5bc019400 con 0x7fe5cc07be80 2023-07-20T18:51:08.814 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.317+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/3531472197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5cc061da0 con 0x7fe5cc07be80 2023-07-20T18:51:08.815 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.320+0000 7fe5bb7fe700 1 -- 172.21.15.57:0/3531472197 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fe5bc01abd0 con 0x7fe5cc07be80 2023-07-20T18:51:08.815 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.393+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/3531472197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fe5cc19f460 con 0x7fe5cc07be80 2023-07-20T18:51:08.815 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.396+0000 7fe5bb7fe700 1 -- 172.21.15.57:0/3531472197 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe5bc01ad70 con 0x7fe5cc07be80 2023-07-20T18:51:08.815 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.397+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/3531472197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 msgr2=0x7fe5cc19bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:08.815 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.397+0000 7fe5d0cdc700 1 --2- 172.21.15.57:0/3531472197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc19bec0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009130 tx=0x7fe5bc011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:08.815 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.397+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/3531472197 shutdown_connections 2023-07-20T18:51:08.815 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.397+0000 7fe5d0cdc700 1 --2- 172.21.15.57:0/3531472197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5cc07be80 0x7fe5cc19bec0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:08.816 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.397+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/3531472197 >> 172.21.15.57:0/3531472197 conn(0x7fe5cc101610 msgr2=0x7fe5cc102400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:08.816 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.397+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/3531472197 shutdown_connections 2023-07-20T18:51:08.816 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:08.397+0000 7fe5d0cdc700 1 -- 172.21.15.57:0/3531472197 wait complete. 2023-07-20T18:51:08.816 INFO:teuthology.orchestra.run.smithi057.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-20T18:51:08.831 INFO:teuthology.orchestra.run.smithi057.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-20T18:51:08.832 INFO:teuthology.orchestra.run.smithi057.stdout:Creating mgr... 2023-07-20T18:51:08.832 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying port 9283 ... 2023-07-20T18:51:08.832 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying port 8765 ... 2023-07-20T18:51:09.206 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T18:51:09.206 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Failed to reset failed state of unit ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.x.service: Unit ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.x.service not loaded. 2023-07-20T18:51:09.418 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-38a58c3c-272e-11ee-9b34-001a4aab830c.target.wants/ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@.service. 2023-07-20T18:51:09.471 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:09 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3531472197' entity='client.admin' 2023-07-20T18:51:09.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:09 smithi057 systemd[1]: Starting Ceph mgr.x for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:51:10.198 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:09 smithi057 podman[126568]: 2023-07-20T18:51:10.198 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:10 smithi057 bash[126568]: 5ffd14b97a9ec4957a63b49d60ea599650c7a4a2a25f16101054019627b984fb 2023-07-20T18:51:10.223 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T18:51:10.223 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout disabled 2023-07-20T18:51:10.242 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T18:51:10.242 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2023-07-20T18:51:10.243 INFO:teuthology.orchestra.run.smithi057.stdout:firewalld.service is not enabled 2023-07-20T18:51:10.243 INFO:teuthology.orchestra.run.smithi057.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T18:51:10.264 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T18:51:10.264 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout disabled 2023-07-20T18:51:10.283 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T18:51:10.283 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2023-07-20T18:51:10.284 INFO:teuthology.orchestra.run.smithi057.stdout:firewalld.service is not enabled 2023-07-20T18:51:10.284 INFO:teuthology.orchestra.run.smithi057.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-20T18:51:10.284 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mgr to start... 2023-07-20T18:51:10.284 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mgr... 2023-07-20T18:51:10.460 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:10 smithi057 systemd[1]: Started Ceph mgr.x for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:51:10.460 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:10 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:10.314+0000 7f69ad39b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T18:51:10.460 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:10 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:10.423+0000 7f69ad39b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T18:51:10.721 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:10 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:10.552+0000 7f69ad39b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T18:51:10.993 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:10 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:10.687+0000 7f69ad39b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T18:51:10.994 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:10 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:10.927+0000 7f69ad39b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T18:51:11.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:11.369 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2023-07-20T18:51:11.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsid": "38a58c3c-272e-11ee-9b34-001a4aab830c", 2023-07-20T18:51:11.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T18:51:11.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T18:51:11.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T18:51:11.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T18:51:11.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:11.371 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T18:51:11.371 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T18:51:11.371 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 0 2023-07-20T18:51:11.371 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2023-07-20T18:51:11.371 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T18:51:11.371 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T18:51:11.371 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2023-07-20T18:51:11.372 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2023-07-20T18:51:11.373 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T18:51:11.373 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:11.373 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T18:51:11.373 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T18:51:11.374 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:11.374 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T18:51:11.374 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:11.375 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:10 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1431461123' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T18:51:11.384 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T18:51:11.384 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T18:51:11.385 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T18:51:11.385 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T18:51:11.385 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T18:51:11.385 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T18:51:11.385 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:11.385 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T18:51:11.386 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T18:51:11.386 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T18:51:11.386 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T18:51:11.386 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T18:51:11.386 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T18:51:11.386 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T18:51:11.386 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T18:51:11.386 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T18:51:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T18:51:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T18:51:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T18:51:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T18:51:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T18:51:11.388 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T18:51:11.388 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T18:51:11.388 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T18:51:11.388 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T18:51:11.388 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T18:51:11.388 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2023-07-20T18:51:11.389 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T18:51:11.389 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:11.389 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T18:51:11.389 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:11.389 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T18:51:01.246115+0000", 2023-07-20T18:51:11.389 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T18:51:11.389 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:11.389 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T18:51:11.390 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2023-07-20T18:51:11.390 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.816+0000 7f7fdaa44700 1 Processor -- start 2023-07-20T18:51:11.390 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.817+0000 7f7fdaa44700 1 -- start start 2023-07-20T18:51:11.390 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.817+0000 7f7fdaa44700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd407a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:11.390 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.817+0000 7f7fdaa44700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd407a8b0 con 0x7f7fd407be80 2023-07-20T18:51:11.390 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.817+0000 7f7fd3fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd407a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:11.391 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.817+0000 7f7fd3fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd407a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53416/0 (socket says 172.21.15.57:53416) 2023-07-20T18:51:11.391 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.817+0000 7f7fd3fff700 1 -- 172.21.15.57:0/3981584234 learned_addr learned my addr 172.21.15.57:0/3981584234 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:11.391 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.818+0000 7f7fd3fff700 1 -- 172.21.15.57:0/3981584234 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd407a9f0 con 0x7f7fd407be80 2023-07-20T18:51:11.391 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.818+0000 7f7fd3fff700 1 --2- 172.21.15.57:0/3981584234 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd407a2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7fc40098a0 tx=0x7f7fc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50c5ea86a07013b server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:11.391 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.818+0000 7f7fd2ffd700 1 -- 172.21.15.57:0/3981584234 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fc4004160 con 0x7f7fd407be80 2023-07-20T18:51:11.391 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.818+0000 7f7fd2ffd700 1 -- 172.21.15.57:0/3981584234 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f7fc40042c0 con 0x7f7fd407be80 2023-07-20T18:51:11.391 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.818+0000 7f7fd2ffd700 1 -- 172.21.15.57:0/3981584234 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fc4011ac0 con 0x7f7fd407be80 2023-07-20T18:51:11.392 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.819+0000 7f7fdaa44700 1 -- 172.21.15.57:0/3981584234 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 msgr2=0x7f7fd407a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:11.392 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.819+0000 7f7fdaa44700 1 --2- 172.21.15.57:0/3981584234 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd407a2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7fc40098a0 tx=0x7f7fc4009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:11.392 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.819+0000 7f7fdaa44700 1 -- 172.21.15.57:0/3981584234 shutdown_connections 2023-07-20T18:51:11.392 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.819+0000 7f7fdaa44700 1 --2- 172.21.15.57:0/3981584234 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd407a2e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:11.392 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.819+0000 7f7fdaa44700 1 -- 172.21.15.57:0/3981584234 >> 172.21.15.57:0/3981584234 conn(0x7f7fd4101610 msgr2=0x7f7fd4103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:11.392 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.819+0000 7f7fdaa44700 1 -- 172.21.15.57:0/3981584234 shutdown_connections 2023-07-20T18:51:11.392 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.819+0000 7f7fdaa44700 1 -- 172.21.15.57:0/3981584234 wait complete. 2023-07-20T18:51:11.393 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.821+0000 7f7fdaa44700 1 Processor -- start 2023-07-20T18:51:11.393 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.821+0000 7f7fdaa44700 1 -- start start 2023-07-20T18:51:11.393 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.821+0000 7f7fdaa44700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd419bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:11.393 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.821+0000 7f7fdaa44700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd419c400 con 0x7f7fd407be80 2023-07-20T18:51:11.393 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.822+0000 7f7fd3fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd419bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:11.393 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.822+0000 7f7fd3fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd419bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53432/0 (socket says 172.21.15.57:53432) 2023-07-20T18:51:11.393 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.822+0000 7f7fd3fff700 1 -- 172.21.15.57:0/1431461123 learned_addr learned my addr 172.21.15.57:0/1431461123 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:11.394 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.822+0000 7f7fd3fff700 1 -- 172.21.15.57:0/1431461123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc4009550 con 0x7f7fd407be80 2023-07-20T18:51:11.394 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.822+0000 7f7fd3fff700 1 --2- 172.21.15.57:0/1431461123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd419bec0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4009130 tx=0x7f7fc4011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:11.394 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.823+0000 7f7fd17fa700 1 -- 172.21.15.57:0/1431461123 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fc4003aa0 con 0x7f7fd407be80 2023-07-20T18:51:11.394 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.823+0000 7f7fd17fa700 1 -- 172.21.15.57:0/1431461123 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f7fc4003c00 con 0x7f7fd407be80 2023-07-20T18:51:11.394 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.823+0000 7f7fdaa44700 1 -- 172.21.15.57:0/1431461123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd419c600 con 0x7f7fd407be80 2023-07-20T18:51:11.394 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.823+0000 7f7fd17fa700 1 -- 172.21.15.57:0/1431461123 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fc4024800 con 0x7f7fd407be80 2023-07-20T18:51:11.394 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.823+0000 7f7fdaa44700 1 -- 172.21.15.57:0/1431461123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd419ca20 con 0x7f7fd407be80 2023-07-20T18:51:11.394 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.823+0000 7f7fd17fa700 1 -- 172.21.15.57:0/1431461123 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7fc402d3f0 con 0x7f7fd407be80 2023-07-20T18:51:11.395 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.823+0000 7f7fd17fa700 1 -- 172.21.15.57:0/1431461123 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7fc40193c0 con 0x7f7fd407be80 2023-07-20T18:51:11.395 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.824+0000 7f7fdaa44700 1 -- 172.21.15.57:0/1431461123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd410c4c0 con 0x7f7fd407be80 2023-07-20T18:51:11.395 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.827+0000 7f7fd17fa700 1 -- 172.21.15.57:0/1431461123 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f7fc401abd0 con 0x7f7fd407be80 2023-07-20T18:51:11.395 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.904+0000 7f7fdaa44700 1 -- 172.21.15.57:0/1431461123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7fd419fab0 con 0x7f7fd407be80 2023-07-20T18:51:11.395 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.905+0000 7f7fd17fa700 1 -- 172.21.15.57:0/1431461123 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f7fc401a430 con 0x7f7fd407be80 2023-07-20T18:51:11.395 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.906+0000 7f7fdaa44700 1 -- 172.21.15.57:0/1431461123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 msgr2=0x7f7fd419bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:11.395 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.906+0000 7f7fdaa44700 1 --2- 172.21.15.57:0/1431461123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd419bec0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4009130 tx=0x7f7fc4011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:11.396 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.906+0000 7f7fdaa44700 1 -- 172.21.15.57:0/1431461123 shutdown_connections 2023-07-20T18:51:11.396 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.906+0000 7f7fdaa44700 1 --2- 172.21.15.57:0/1431461123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fd407be80 0x7f7fd419bec0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:11.396 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.906+0000 7f7fdaa44700 1 -- 172.21.15.57:0/1431461123 >> 172.21.15.57:0/1431461123 conn(0x7f7fd4101610 msgr2=0x7f7fd4102400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:11.396 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.906+0000 7f7fdaa44700 1 -- 172.21.15.57:0/1431461123 shutdown_connections 2023-07-20T18:51:11.396 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:10.906+0000 7f7fdaa44700 1 -- 172.21.15.57:0/1431461123 wait complete. 2023-07-20T18:51:11.396 INFO:teuthology.orchestra.run.smithi057.stdout:mgr not available, waiting (1/15)... 2023-07-20T18:51:11.692 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:11.368+0000 7f69ad39b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T18:51:11.692 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:11.489+0000 7f69ad39b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T18:51:11.971 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:11.655+0000 7f69ad39b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T18:51:11.971 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:11.784+0000 7f69ad39b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T18:51:12.721 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:12 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:12.349+0000 7f69ad39b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T18:51:13.471 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:13 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:13.161+0000 7f69ad39b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T18:51:13.471 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:13 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:13.378+0000 7f69ad39b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T18:51:14.465 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:14 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2311965614' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T18:51:14.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:14.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2023-07-20T18:51:14.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsid": "38a58c3c-272e-11ee-9b34-001a4aab830c", 2023-07-20T18:51:14.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T18:51:14.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T18:51:14.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T18:51:14.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T18:51:14.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:14.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T18:51:14.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T18:51:14.642 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 0 2023-07-20T18:51:14.642 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2023-07-20T18:51:14.642 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T18:51:14.643 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T18:51:14.643 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2023-07-20T18:51:14.643 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2023-07-20T18:51:14.643 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T18:51:14.643 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:14.643 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T18:51:14.643 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T18:51:14.643 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:14.644 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T18:51:14.644 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:14.644 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T18:51:14.644 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T18:51:14.644 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T18:51:14.644 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T18:51:14.644 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T18:51:14.645 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T18:51:14.645 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:14.645 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T18:51:14.645 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T18:51:14.645 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T18:51:14.645 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T18:51:14.645 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T18:51:14.645 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T18:51:14.646 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T18:51:14.646 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T18:51:14.646 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T18:51:14.646 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:14.646 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T18:51:14.646 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:14.646 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T18:51:14.646 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T18:51:14.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:14.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T18:51:14.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T18:51:14.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T18:51:14.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T18:51:14.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T18:51:14.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T18:51:14.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T18:51:14.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2023-07-20T18:51:14.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T18:51:14.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:14.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T18:51:14.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:14.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T18:51:01.246115+0000", 2023-07-20T18:51:14.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T18:51:14.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:14.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T18:51:14.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2023-07-20T18:51:14.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.913+0000 7f90825fb700 1 Processor -- start 2023-07-20T18:51:14.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.913+0000 7f90825fb700 1 -- start start 2023-07-20T18:51:14.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.913+0000 7f90825fb700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:14.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.913+0000 7f90825fb700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c106a60 con 0x7f907c106070 2023-07-20T18:51:14.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.914+0000 7f907bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:14.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.914+0000 7f907bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53442/0 (socket says 172.21.15.57:53442) 2023-07-20T18:51:14.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.914+0000 7f907bfff700 1 -- 172.21.15.57:0/2156230560 learned_addr learned my addr 172.21.15.57:0/2156230560 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:14.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.914+0000 7f907bfff700 1 -- 172.21.15.57:0/2156230560 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c107270 con 0x7f907c106070 2023-07-20T18:51:14.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.914+0000 7f907bfff700 1 --2- 172.21.15.57:0/2156230560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c106490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f906c0098a0 tx=0x7f906c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62de59325bec0c1c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:14.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.915+0000 7f907affd700 1 -- 172.21.15.57:0/2156230560 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f906c004160 con 0x7f907c106070 2023-07-20T18:51:14.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.915+0000 7f907affd700 1 -- 172.21.15.57:0/2156230560 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f906c0042c0 con 0x7f907c106070 2023-07-20T18:51:14.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.915+0000 7f907affd700 1 -- 172.21.15.57:0/2156230560 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f906c011ac0 con 0x7f907c106070 2023-07-20T18:51:14.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.915+0000 7f90825fb700 1 -- 172.21.15.57:0/2156230560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 msgr2=0x7f907c106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:14.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.915+0000 7f90825fb700 1 --2- 172.21.15.57:0/2156230560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c106490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f906c0098a0 tx=0x7f906c009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:14.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.916+0000 7f90825fb700 1 -- 172.21.15.57:0/2156230560 shutdown_connections 2023-07-20T18:51:14.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.916+0000 7f90825fb700 1 --2- 172.21.15.57:0/2156230560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c106490 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:14.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.916+0000 7f90825fb700 1 -- 172.21.15.57:0/2156230560 >> 172.21.15.57:0/2156230560 conn(0x7f907c1015f0 msgr2=0x7f907c103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:14.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.916+0000 7f90825fb700 1 -- 172.21.15.57:0/2156230560 shutdown_connections 2023-07-20T18:51:14.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.916+0000 7f90825fb700 1 -- 172.21.15.57:0/2156230560 wait complete. 2023-07-20T18:51:14.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.917+0000 7f90825fb700 1 Processor -- start 2023-07-20T18:51:14.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.917+0000 7f90825fb700 1 -- start start 2023-07-20T18:51:14.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.917+0000 7f90825fb700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:14.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.917+0000 7f90825fb700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c19a2a0 con 0x7f907c106070 2023-07-20T18:51:14.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.917+0000 7f907bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:14.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.917+0000 7f907bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53456/0 (socket says 172.21.15.57:53456) 2023-07-20T18:51:14.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.917+0000 7f907bfff700 1 -- 172.21.15.57:0/2311965614 learned_addr learned my addr 172.21.15.57:0/2311965614 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:14.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.918+0000 7f907bfff700 1 -- 172.21.15.57:0/2311965614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906c009550 con 0x7f907c106070 2023-07-20T18:51:14.653 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.918+0000 7f907bfff700 1 --2- 172.21.15.57:0/2311965614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c199d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f906c000c00 tx=0x7f906c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:14.653 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.918+0000 7f90797fa700 1 -- 172.21.15.57:0/2311965614 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f906c003aa0 con 0x7f907c106070 2023-07-20T18:51:14.653 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.918+0000 7f90797fa700 1 -- 172.21.15.57:0/2311965614 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f906c003c00 con 0x7f907c106070 2023-07-20T18:51:14.653 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.918+0000 7f90825fb700 1 -- 172.21.15.57:0/2311965614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907c19a4a0 con 0x7f907c106070 2023-07-20T18:51:14.653 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.918+0000 7f90797fa700 1 -- 172.21.15.57:0/2311965614 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f906c0248a0 con 0x7f907c106070 2023-07-20T18:51:14.654 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.919+0000 7f90825fb700 1 -- 172.21.15.57:0/2311965614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907c19a8c0 con 0x7f907c106070 2023-07-20T18:51:14.654 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.919+0000 7f90797fa700 1 -- 172.21.15.57:0/2311965614 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f906c02d3f0 con 0x7f907c106070 2023-07-20T18:51:14.654 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.919+0000 7f90797fa700 1 -- 172.21.15.57:0/2311965614 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f906c0194a0 con 0x7f907c106070 2023-07-20T18:51:14.654 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.920+0000 7f90825fb700 1 -- 172.21.15.57:0/2311965614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907c061da0 con 0x7f907c106070 2023-07-20T18:51:14.654 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.922+0000 7f90797fa700 1 -- 172.21.15.57:0/2311965614 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f906c01ac60 con 0x7f907c106070 2023-07-20T18:51:14.654 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:13.999+0000 7f90825fb700 1 -- 172.21.15.57:0/2311965614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f907c19d300 con 0x7f907c106070 2023-07-20T18:51:14.654 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:14.000+0000 7f90797fa700 1 -- 172.21.15.57:0/2311965614 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f906c01ae00 con 0x7f907c106070 2023-07-20T18:51:14.654 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:14.001+0000 7f90825fb700 1 -- 172.21.15.57:0/2311965614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 msgr2=0x7f907c199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:14.655 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:14.001+0000 7f90825fb700 1 --2- 172.21.15.57:0/2311965614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c199d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f906c000c00 tx=0x7f906c011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:14.655 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:14.001+0000 7f90825fb700 1 -- 172.21.15.57:0/2311965614 shutdown_connections 2023-07-20T18:51:14.655 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:14.001+0000 7f90825fb700 1 --2- 172.21.15.57:0/2311965614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f907c106070 0x7f907c199d60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:14.655 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:14.001+0000 7f90825fb700 1 -- 172.21.15.57:0/2311965614 >> 172.21.15.57:0/2311965614 conn(0x7f907c1015f0 msgr2=0x7f907c1023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:14.655 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:14.001+0000 7f90825fb700 1 -- 172.21.15.57:0/2311965614 shutdown_connections 2023-07-20T18:51:14.655 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:14.001+0000 7f90825fb700 1 -- 172.21.15.57:0/2311965614 wait complete. 2023-07-20T18:51:14.655 INFO:teuthology.orchestra.run.smithi057.stdout:mgr not available, waiting (2/15)... 2023-07-20T18:51:14.722 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:14 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:14.428+0000 7f69ad39b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T18:51:14.723 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:14 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:14.556+0000 7f69ad39b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T18:51:15.203 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:14 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:14.788+0000 7f69ad39b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T18:51:15.471 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:15 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:15.166+0000 7f69ad39b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T18:51:15.971 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:15 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:15.453+0000 7f69ad39b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T18:51:15.971 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:15 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:15.653+0000 7f69ad39b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T18:51:16.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:16 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:16.006+0000 7f69ad39b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T18:51:17.445 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:16 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:16.974+0000 7f69ad39b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T18:51:17.446 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:17 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:17.154+0000 7f69ad39b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T18:51:17.446 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:17 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:17.408+0000 7f69ad39b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T18:51:17.721 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:17 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:17.561+0000 7f69ad39b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T18:51:18.222 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:17 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:17.849+0000 7f69ad39b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T18:51:19.471 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:18 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:18.960+0000 7f69ad39b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T18:51:19.472 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: Activating manager daemon x 2023-07-20T18:51:19.472 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: mgrmap e2: x(active, starting, since 0.00394036s) 2023-07-20T18:51:19.473 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: from='mgr.14100 172.21.15.57:0/1529574040' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T18:51:19.473 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: from='mgr.14100 172.21.15.57:0/1529574040' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T18:51:19.473 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: from='mgr.14100 172.21.15.57:0/1529574040' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T18:51:19.473 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: from='mgr.14100 172.21.15.57:0/1529574040' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:51:19.473 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: from='mgr.14100 172.21.15.57:0/1529574040' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T18:51:19.473 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: Manager daemon x is now available 2023-07-20T18:51:19.473 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: from='mgr.14100 172.21.15.57:0/1529574040' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T18:51:19.474 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: from='mgr.14100 172.21.15.57:0/1529574040' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T18:51:19.474 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: from='mgr.14100 172.21.15.57:0/1529574040' entity='mgr.x' 2023-07-20T18:51:19.474 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: from='mgr.14100 172.21.15.57:0/1529574040' entity='mgr.x' 2023-07-20T18:51:19.474 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:19 smithi057 ceph-mon[126340]: from='mgr.14100 172.21.15.57:0/1529574040' entity='mgr.x' 2023-07-20T18:51:21.471 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:20 smithi057 ceph-mon[126340]: mgrmap e3: x(active, since 1.00957s) 2023-07-20T18:51:22.471 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:21 smithi057 ceph-mon[126340]: mgrmap e4: x(active, since 2s) 2023-07-20T18:51:23.786 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:23.787 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2023-07-20T18:51:23.787 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsid": "38a58c3c-272e-11ee-9b34-001a4aab830c", 2023-07-20T18:51:23.787 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T18:51:23.787 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T18:51:23.787 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T18:51:23.787 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T18:51:23.789 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:23.789 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T18:51:23.790 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T18:51:23.790 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 0 2023-07-20T18:51:23.790 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2023-07-20T18:51:23.790 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T18:51:23.790 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T18:51:23.790 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2023-07-20T18:51:23.790 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2023-07-20T18:51:23.790 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T18:51:23.791 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:23.791 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T18:51:23.791 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T18:51:23.791 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:23.791 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T18:51:23.791 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:23.791 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T18:51:23.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T18:51:23.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T18:51:23.798 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T18:51:23.799 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T18:51:23.799 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T18:51:23.799 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:23.799 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T18:51:23.799 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T18:51:23.799 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T18:51:23.799 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T18:51:23.800 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T18:51:23.800 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T18:51:23.800 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T18:51:23.800 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T18:51:23.800 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T18:51:23.800 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:23.800 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T18:51:23.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:23.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T18:51:23.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T18:51:23.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:23.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T18:51:23.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T18:51:23.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T18:51:23.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T18:51:23.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T18:51:23.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T18:51:23.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T18:51:23.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2023-07-20T18:51:23.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T18:51:23.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:23.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T18:51:23.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T18:51:23.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T18:51:01.246115+0000", 2023-07-20T18:51:23.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T18:51:23.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2023-07-20T18:51:23.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T18:51:23.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2023-07-20T18:51:23.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.945+0000 7f0a2811a700 1 Processor -- start 2023-07-20T18:51:23.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.945+0000 7f0a2811a700 1 -- start start 2023-07-20T18:51:23.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.945+0000 7f0a2811a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2007a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:23.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.945+0000 7f0a2811a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2007a8b0 con 0x7f0a2007be80 2023-07-20T18:51:23.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.946+0000 7f0a25eb6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2007a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:23.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.946+0000 7f0a25eb6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2007a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59206/0 (socket says 172.21.15.57:59206) 2023-07-20T18:51:23.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.946+0000 7f0a25eb6700 1 -- 172.21.15.57:0/3384795508 learned_addr learned my addr 172.21.15.57:0/3384795508 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:23.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.946+0000 7f0a25eb6700 1 -- 172.21.15.57:0/3384795508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2007a9f0 con 0x7f0a2007be80 2023-07-20T18:51:23.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.946+0000 7f0a25eb6700 1 --2- 172.21.15.57:0/3384795508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2007a2e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c0098a0 tx=0x7f0a1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22dc601cd90b47a server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:23.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.946+0000 7f0a24eb4700 1 -- 172.21.15.57:0/3384795508 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a1c004160 con 0x7f0a2007be80 2023-07-20T18:51:23.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.947+0000 7f0a24eb4700 1 -- 172.21.15.57:0/3384795508 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f0a1c0042c0 con 0x7f0a2007be80 2023-07-20T18:51:23.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.947+0000 7f0a24eb4700 1 -- 172.21.15.57:0/3384795508 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a1c011ac0 con 0x7f0a2007be80 2023-07-20T18:51:23.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.947+0000 7f0a2811a700 1 -- 172.21.15.57:0/3384795508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 msgr2=0x7f0a2007a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:23.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.947+0000 7f0a2811a700 1 --2- 172.21.15.57:0/3384795508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2007a2e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c0098a0 tx=0x7f0a1c009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:23.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.947+0000 7f0a2811a700 1 -- 172.21.15.57:0/3384795508 shutdown_connections 2023-07-20T18:51:23.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.947+0000 7f0a2811a700 1 --2- 172.21.15.57:0/3384795508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2007a2e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:23.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.947+0000 7f0a2811a700 1 -- 172.21.15.57:0/3384795508 >> 172.21.15.57:0/3384795508 conn(0x7f0a20101610 msgr2=0x7f0a20103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:23.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.947+0000 7f0a2811a700 1 -- 172.21.15.57:0/3384795508 shutdown_connections 2023-07-20T18:51:23.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.947+0000 7f0a2811a700 1 -- 172.21.15.57:0/3384795508 wait complete. 2023-07-20T18:51:23.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.948+0000 7f0a2811a700 1 Processor -- start 2023-07-20T18:51:23.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.949+0000 7f0a2811a700 1 -- start start 2023-07-20T18:51:23.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.949+0000 7f0a2811a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2019bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:23.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.949+0000 7f0a2811a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2019c400 con 0x7f0a2007be80 2023-07-20T18:51:23.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.949+0000 7f0a25eb6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2019bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:23.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.949+0000 7f0a25eb6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2019bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59220/0 (socket says 172.21.15.57:59220) 2023-07-20T18:51:23.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.949+0000 7f0a25eb6700 1 -- 172.21.15.57:0/2404806313 learned_addr learned my addr 172.21.15.57:0/2404806313 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:23.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.949+0000 7f0a25eb6700 1 -- 172.21.15.57:0/2404806313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c009550 con 0x7f0a2007be80 2023-07-20T18:51:23.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.950+0000 7f0a25eb6700 1 --2- 172.21.15.57:0/2404806313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2019bec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009130 tx=0x7f0a1c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:23.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.950+0000 7f0a16ffd700 1 -- 172.21.15.57:0/2404806313 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a1c003aa0 con 0x7f0a2007be80 2023-07-20T18:51:23.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.950+0000 7f0a16ffd700 1 -- 172.21.15.57:0/2404806313 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f0a1c003c00 con 0x7f0a2007be80 2023-07-20T18:51:23.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.950+0000 7f0a16ffd700 1 -- 172.21.15.57:0/2404806313 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a1c024800 con 0x7f0a2007be80 2023-07-20T18:51:23.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.950+0000 7f0a2811a700 1 -- 172.21.15.57:0/2404806313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2019c600 con 0x7f0a2007be80 2023-07-20T18:51:23.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.950+0000 7f0a2811a700 1 -- 172.21.15.57:0/2404806313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2019ca20 con 0x7f0a2007be80 2023-07-20T18:51:23.808 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.951+0000 7f0a16ffd700 1 -- 172.21.15.57:0/2404806313 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f0a1c01abd0 con 0x7f0a2007be80 2023-07-20T18:51:23.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.951+0000 7f0a2811a700 1 -- 172.21.15.57:0/2404806313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2010c4c0 con 0x7f0a2007be80 2023-07-20T18:51:23.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.952+0000 7f0a16ffd700 1 --2- 172.21.15.57:0/2404806313 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f0a0c038780 0x7f0a0c03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:23.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.952+0000 7f0a16ffd700 1 -- 172.21.15.57:0/2404806313 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0a1c04c4c0 con 0x7f0a2007be80 2023-07-20T18:51:23.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.952+0000 7f0a256b5700 1 --2- 172.21.15.57:0/2404806313 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f0a0c038780 0x7f0a0c03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:23.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.953+0000 7f0a256b5700 1 --2- 172.21.15.57:0/2404806313 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f0a0c038780 0x7f0a0c03ac40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0a10006fd0 tx=0x7f0a10006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:23.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:22.958+0000 7f0a16ffd700 1 -- 172.21.15.57:0/2404806313 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0a1c024ac0 con 0x7f0a2007be80 2023-07-20T18:51:23.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.242+0000 7f0a2811a700 1 -- 172.21.15.57:0/2404806313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0a2010c6b0 con 0x7f0a2007be80 2023-07-20T18:51:23.809 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.242+0000 7f0a16ffd700 1 -- 172.21.15.57:0/2404806313 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f0a1c018350 con 0x7f0a2007be80 2023-07-20T18:51:23.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.245+0000 7f0a2811a700 1 -- 172.21.15.57:0/2404806313 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f0a0c038780 msgr2=0x7f0a0c03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:23.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.245+0000 7f0a2811a700 1 --2- 172.21.15.57:0/2404806313 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f0a0c038780 0x7f0a0c03ac40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0a10006fd0 tx=0x7f0a10006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:23.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.245+0000 7f0a2811a700 1 -- 172.21.15.57:0/2404806313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 msgr2=0x7f0a2019bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:23.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.245+0000 7f0a2811a700 1 --2- 172.21.15.57:0/2404806313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2019bec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009130 tx=0x7f0a1c011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:23.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.245+0000 7f0a2811a700 1 -- 172.21.15.57:0/2404806313 shutdown_connections 2023-07-20T18:51:23.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.245+0000 7f0a2811a700 1 --2- 172.21.15.57:0/2404806313 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f0a0c038780 0x7f0a0c03ac40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:23.810 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.245+0000 7f0a2811a700 1 --2- 172.21.15.57:0/2404806313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2007be80 0x7f0a2019bec0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:23.811 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.245+0000 7f0a2811a700 1 -- 172.21.15.57:0/2404806313 >> 172.21.15.57:0/2404806313 conn(0x7f0a20101610 msgr2=0x7f0a20102400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:23.811 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.245+0000 7f0a2811a700 1 -- 172.21.15.57:0/2404806313 shutdown_connections 2023-07-20T18:51:23.811 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:23.245+0000 7f0a2811a700 1 -- 172.21.15.57:0/2404806313 wait complete. 2023-07-20T18:51:23.811 INFO:teuthology.orchestra.run.smithi057.stdout:mgr is available 2023-07-20T18:51:24.073 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:23 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2404806313' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T18:51:24.971 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:24 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2298974847' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T18:51:25.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:25.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T18:51:25.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout fsid = 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:51:25.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T18:51:25.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T18:51:25.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T18:51:25.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T18:51:25.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:25.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T18:51:25.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T18:51:25.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2023-07-20T18:51:25.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T18:51:25.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T18:51:25.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T18:51:25.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T18:51:25.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T18:51:25.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T18:51:25.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.311+0000 7f7c6f6a9700 1 Processor -- start 2023-07-20T18:51:25.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.311+0000 7f7c6f6a9700 1 -- start start 2023-07-20T18:51:25.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.311+0000 7f7c6f6a9700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c68106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:25.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.311+0000 7f7c6f6a9700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68106a60 con 0x7f7c68106070 2023-07-20T18:51:25.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.311+0000 7f7c6d445700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c68106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:25.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.311+0000 7f7c6d445700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c68106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59230/0 (socket says 172.21.15.57:59230) 2023-07-20T18:51:25.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.311+0000 7f7c6d445700 1 -- 172.21.15.57:0/2197958422 learned_addr learned my addr 172.21.15.57:0/2197958422 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:25.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.312+0000 7f7c6d445700 1 -- 172.21.15.57:0/2197958422 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c68107270 con 0x7f7c68106070 2023-07-20T18:51:25.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.312+0000 7f7c6d445700 1 --2- 172.21.15.57:0/2197958422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c68106490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7c640098a0 tx=0x7f7c64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5931dac9cb5ffe7b server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:25.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.312+0000 7f7c5ffff700 1 -- 172.21.15.57:0/2197958422 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c64004160 con 0x7f7c68106070 2023-07-20T18:51:25.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.312+0000 7f7c5ffff700 1 -- 172.21.15.57:0/2197958422 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f7c640042c0 con 0x7f7c68106070 2023-07-20T18:51:25.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.312+0000 7f7c5ffff700 1 -- 172.21.15.57:0/2197958422 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c64011ac0 con 0x7f7c68106070 2023-07-20T18:51:25.083 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.313+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2197958422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 msgr2=0x7f7c68106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:25.084 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.313+0000 7f7c6f6a9700 1 --2- 172.21.15.57:0/2197958422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c68106490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7c640098a0 tx=0x7f7c64009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:25.084 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.313+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2197958422 shutdown_connections 2023-07-20T18:51:25.084 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.313+0000 7f7c6f6a9700 1 --2- 172.21.15.57:0/2197958422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c68106490 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:25.084 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.313+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2197958422 >> 172.21.15.57:0/2197958422 conn(0x7f7c68101610 msgr2=0x7f7c68103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:25.085 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.313+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2197958422 shutdown_connections 2023-07-20T18:51:25.085 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.313+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2197958422 wait complete. 2023-07-20T18:51:25.085 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.314+0000 7f7c6f6a9700 1 Processor -- start 2023-07-20T18:51:25.085 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.314+0000 7f7c6f6a9700 1 -- start start 2023-07-20T18:51:25.085 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.315+0000 7f7c6f6a9700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c6819bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:25.085 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.315+0000 7f7c6f6a9700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6819c470 con 0x7f7c68106070 2023-07-20T18:51:25.086 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.315+0000 7f7c6d445700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c6819bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:25.086 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.315+0000 7f7c6d445700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c6819bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59236/0 (socket says 172.21.15.57:59236) 2023-07-20T18:51:25.086 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.315+0000 7f7c6d445700 1 -- 172.21.15.57:0/2298974847 learned_addr learned my addr 172.21.15.57:0/2298974847 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:25.086 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.315+0000 7f7c6d445700 1 -- 172.21.15.57:0/2298974847 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c64009550 con 0x7f7c68106070 2023-07-20T18:51:25.086 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.315+0000 7f7c6d445700 1 --2- 172.21.15.57:0/2298974847 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c6819bf30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7c64000c00 tx=0x7f7c64011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:25.086 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.316+0000 7f7c5e7fc700 1 -- 172.21.15.57:0/2298974847 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c64003aa0 con 0x7f7c68106070 2023-07-20T18:51:25.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.316+0000 7f7c5e7fc700 1 -- 172.21.15.57:0/2298974847 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f7c64003c00 con 0x7f7c68106070 2023-07-20T18:51:25.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.316+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2298974847 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6819c670 con 0x7f7c68106070 2023-07-20T18:51:25.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.316+0000 7f7c5e7fc700 1 -- 172.21.15.57:0/2298974847 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c640248a0 con 0x7f7c68106070 2023-07-20T18:51:25.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.316+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2298974847 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6819ca90 con 0x7f7c68106070 2023-07-20T18:51:25.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.317+0000 7f7c5e7fc700 1 -- 172.21.15.57:0/2298974847 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f7c6401ac60 con 0x7f7c68106070 2023-07-20T18:51:25.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.317+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2298974847 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c68061da0 con 0x7f7c68106070 2023-07-20T18:51:25.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.317+0000 7f7c5e7fc700 1 --2- 172.21.15.57:0/2298974847 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f7c54038780 0x7f7c5403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:25.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.317+0000 7f7c5e7fc700 1 -- 172.21.15.57:0/2298974847 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7c6404c380 con 0x7f7c68106070 2023-07-20T18:51:25.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.318+0000 7f7c6cc44700 1 --2- 172.21.15.57:0/2298974847 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f7c54038780 0x7f7c5403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:25.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.318+0000 7f7c6cc44700 1 --2- 172.21.15.57:0/2298974847 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f7c54038780 0x7f7c5403ac40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7c58006fd0 tx=0x7f7c58006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:25.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.323+0000 7f7c5e7fc700 1 -- 172.21.15.57:0/2298974847 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7c64018530 con 0x7f7c68106070 2023-07-20T18:51:25.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.525+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2298974847 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f7c6819f980 con 0x7f7c68106070 2023-07-20T18:51:25.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.526+0000 7f7c5e7fc700 1 -- 172.21.15.57:0/2298974847 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f7c64018350 con 0x7f7c68106070 2023-07-20T18:51:25.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.528+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2298974847 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f7c54038780 msgr2=0x7f7c5403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:25.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.528+0000 7f7c6f6a9700 1 --2- 172.21.15.57:0/2298974847 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f7c54038780 0x7f7c5403ac40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7c58006fd0 tx=0x7f7c58006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:25.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.528+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2298974847 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 msgr2=0x7f7c6819bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:25.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.528+0000 7f7c6f6a9700 1 --2- 172.21.15.57:0/2298974847 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c6819bf30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7c64000c00 tx=0x7f7c64011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:25.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.528+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2298974847 shutdown_connections 2023-07-20T18:51:25.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.528+0000 7f7c6f6a9700 1 --2- 172.21.15.57:0/2298974847 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7f7c54038780 0x7f7c5403ac40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:25.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.528+0000 7f7c6f6a9700 1 --2- 172.21.15.57:0/2298974847 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c68106070 0x7f7c6819bf30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:25.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.528+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2298974847 >> 172.21.15.57:0/2298974847 conn(0x7f7c68101610 msgr2=0x7f7c68102380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:25.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.528+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2298974847 shutdown_connections 2023-07-20T18:51:25.090 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:24.528+0000 7f7c6f6a9700 1 -- 172.21.15.57:0/2298974847 wait complete. 2023-07-20T18:51:25.090 INFO:teuthology.orchestra.run.smithi057.stdout:Enabling cephadm module... 2023-07-20T18:51:26.200 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:25 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1480506781' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-20T18:51:26.201 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:25 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: ignoring --setuser ceph since I am not root 2023-07-20T18:51:26.201 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:25 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: ignoring --setgroup ceph since I am not root 2023-07-20T18:51:26.462 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.571+0000 7fac21212700 1 Processor -- start 2023-07-20T18:51:26.462 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.571+0000 7fac21212700 1 -- start start 2023-07-20T18:51:26.462 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.571+0000 7fac21212700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c07a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:26.463 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.571+0000 7fac21212700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c07a8f0 con 0x7fac1c07bec0 2023-07-20T18:51:26.463 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.572+0000 7fac1ad9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c07a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:26.463 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.572+0000 7fac1ad9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c07a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50920/0 (socket says 172.21.15.57:50920) 2023-07-20T18:51:26.463 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.572+0000 7fac1ad9d700 1 -- 172.21.15.57:0/1812079145 learned_addr learned my addr 172.21.15.57:0/1812079145 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:26.463 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.572+0000 7fac1ad9d700 1 -- 172.21.15.57:0/1812079145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c07aa30 con 0x7fac1c07bec0 2023-07-20T18:51:26.463 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.572+0000 7fac1ad9d700 1 --2- 172.21.15.57:0/1812079145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c07a320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0098a0 tx=0x7fac0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4c45aacbb6058 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:26.463 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.573+0000 7fac19d9b700 1 -- 172.21.15.57:0/1812079145 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac0c004160 con 0x7fac1c07bec0 2023-07-20T18:51:26.464 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.573+0000 7fac19d9b700 1 -- 172.21.15.57:0/1812079145 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fac0c0042c0 con 0x7fac1c07bec0 2023-07-20T18:51:26.464 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.573+0000 7fac19d9b700 1 -- 172.21.15.57:0/1812079145 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac0c011ac0 con 0x7fac1c07bec0 2023-07-20T18:51:26.464 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.573+0000 7fac21212700 1 -- 172.21.15.57:0/1812079145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 msgr2=0x7fac1c07a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:26.464 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.573+0000 7fac21212700 1 --2- 172.21.15.57:0/1812079145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c07a320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0098a0 tx=0x7fac0c009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:26.465 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.573+0000 7fac21212700 1 -- 172.21.15.57:0/1812079145 shutdown_connections 2023-07-20T18:51:26.465 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.573+0000 7fac21212700 1 --2- 172.21.15.57:0/1812079145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c07a320 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:26.465 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.573+0000 7fac21212700 1 -- 172.21.15.57:0/1812079145 >> 172.21.15.57:0/1812079145 conn(0x7fac1c101610 msgr2=0x7fac1c103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:26.466 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.573+0000 7fac21212700 1 -- 172.21.15.57:0/1812079145 shutdown_connections 2023-07-20T18:51:26.466 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.573+0000 7fac21212700 1 -- 172.21.15.57:0/1812079145 wait complete. 2023-07-20T18:51:26.466 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.575+0000 7fac21212700 1 Processor -- start 2023-07-20T18:51:26.466 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.575+0000 7fac21212700 1 -- start start 2023-07-20T18:51:26.466 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.576+0000 7fac21212700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c19bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:26.467 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.576+0000 7fac21212700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c19c440 con 0x7fac1c07bec0 2023-07-20T18:51:26.467 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.576+0000 7fac1ad9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c19bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:26.467 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.576+0000 7fac1ad9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c19bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50932/0 (socket says 172.21.15.57:50932) 2023-07-20T18:51:26.467 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.576+0000 7fac1ad9d700 1 -- 172.21.15.57:0/1480506781 learned_addr learned my addr 172.21.15.57:0/1480506781 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:26.467 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.576+0000 7fac1ad9d700 1 -- 172.21.15.57:0/1480506781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c009550 con 0x7fac1c07bec0 2023-07-20T18:51:26.467 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.576+0000 7fac1ad9d700 1 --2- 172.21.15.57:0/1480506781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c19bf00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fac0c000c00 tx=0x7fac0c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:26.468 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.577+0000 7fac03fff700 1 -- 172.21.15.57:0/1480506781 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac0c003aa0 con 0x7fac1c07bec0 2023-07-20T18:51:26.468 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.577+0000 7fac03fff700 1 -- 172.21.15.57:0/1480506781 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fac0c003c00 con 0x7fac1c07bec0 2023-07-20T18:51:26.468 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.577+0000 7fac21212700 1 -- 172.21.15.57:0/1480506781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c19c640 con 0x7fac1c07bec0 2023-07-20T18:51:26.468 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.577+0000 7fac03fff700 1 -- 172.21.15.57:0/1480506781 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac0c0248a0 con 0x7fac1c07bec0 2023-07-20T18:51:26.468 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.577+0000 7fac21212700 1 -- 172.21.15.57:0/1480506781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c19ca60 con 0x7fac1c07bec0 2023-07-20T18:51:26.469 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.578+0000 7fac03fff700 1 -- 172.21.15.57:0/1480506781 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fac0c01ac60 con 0x7fac1c07bec0 2023-07-20T18:51:26.469 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.578+0000 7fac21212700 1 -- 172.21.15.57:0/1480506781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c061da0 con 0x7fac1c07bec0 2023-07-20T18:51:26.469 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.578+0000 7fac03fff700 1 --2- 172.21.15.57:0/1480506781 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fac040387d0 0x7fac0403ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:26.469 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.578+0000 7fac03fff700 1 -- 172.21.15.57:0/1480506781 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fac0c04c360 con 0x7fac1c07bec0 2023-07-20T18:51:26.469 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.579+0000 7fac1a59c700 1 --2- 172.21.15.57:0/1480506781 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fac040387d0 0x7fac0403ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:26.469 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.579+0000 7fac1a59c700 1 --2- 172.21.15.57:0/1480506781 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fac040387d0 0x7fac0403ac90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fac10006fd0 tx=0x7fac10006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:26.469 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.583+0000 7fac03fff700 1 -- 172.21.15.57:0/1480506781 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fac0c018530 con 0x7fac1c07bec0 2023-07-20T18:51:26.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.842+0000 7fac21212700 1 -- 172.21.15.57:0/1480506781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fac1c19fbe0 con 0x7fac1c07bec0 2023-07-20T18:51:26.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.895+0000 7fac03fff700 1 -- 172.21.15.57:0/1480506781 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fac0c02d770 con 0x7fac1c07bec0 2023-07-20T18:51:26.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.897+0000 7fac03fff700 1 -- 172.21.15.57:0/1480506781 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fac0c02ad60 con 0x7fac1c07bec0 2023-07-20T18:51:26.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.900+0000 7fac21212700 1 -- 172.21.15.57:0/1480506781 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fac040387d0 msgr2=0x7fac0403ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:26.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.900+0000 7fac21212700 1 --2- 172.21.15.57:0/1480506781 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fac040387d0 0x7fac0403ac90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fac10006fd0 tx=0x7fac10006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:26.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.900+0000 7fac21212700 1 -- 172.21.15.57:0/1480506781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 msgr2=0x7fac1c19bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:26.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.900+0000 7fac21212700 1 --2- 172.21.15.57:0/1480506781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c19bf00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fac0c000c00 tx=0x7fac0c011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:26.470 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.900+0000 7fac21212700 1 -- 172.21.15.57:0/1480506781 shutdown_connections 2023-07-20T18:51:26.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.900+0000 7fac21212700 1 --2- 172.21.15.57:0/1480506781 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fac040387d0 0x7fac0403ac90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:26.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.900+0000 7fac21212700 1 --2- 172.21.15.57:0/1480506781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c07bec0 0x7fac1c19bf00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:26.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.900+0000 7fac21212700 1 -- 172.21.15.57:0/1480506781 >> 172.21.15.57:0/1480506781 conn(0x7fac1c101610 msgr2=0x7fac1c1023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:26.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.900+0000 7fac21212700 1 -- 172.21.15.57:0/1480506781 shutdown_connections 2023-07-20T18:51:26.471 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:25.900+0000 7fac21212700 1 -- 172.21.15.57:0/1480506781 wait complete. 2023-07-20T18:51:26.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:26 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:26.163+0000 7fe9f476e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T18:51:26.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:26 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:26.271+0000 7fe9f476e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T18:51:26.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:26 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:26.390+0000 7fe9f476e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T18:51:26.937 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:26 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:26.524+0000 7fe9f476e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T18:51:26.937 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:26 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:26.762+0000 7fe9f476e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T18:51:27.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:26 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1480506781' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-20T18:51:27.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:26 smithi057 ceph-mon[126340]: mgrmap e5: x(active, since 6s) 2023-07-20T18:51:27.465 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:27 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:27.170+0000 7fe9f476e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T18:51:27.465 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:27 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:27.293+0000 7fe9f476e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T18:51:27.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2023-07-20T18:51:27.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-20T18:51:27.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T18:51:27.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T18:51:27.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T18:51:27.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2023-07-20T18:51:27.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.971+0000 7fd346e90700 1 Processor -- start 2023-07-20T18:51:27.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.971+0000 7fd346e90700 1 -- start start 2023-07-20T18:51:27.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.971+0000 7fd346e90700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:27.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.971+0000 7fd346e90700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd340106a60 con 0x7fd340106070 2023-07-20T18:51:27.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.972+0000 7fd344c2c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:27.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.972+0000 7fd344c2c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50962/0 (socket says 172.21.15.57:50962) 2023-07-20T18:51:27.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.972+0000 7fd344c2c700 1 -- 172.21.15.57:0/1446363003 learned_addr learned my addr 172.21.15.57:0/1446363003 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:27.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.972+0000 7fd344c2c700 1 -- 172.21.15.57:0/1446363003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340107270 con 0x7fd340106070 2023-07-20T18:51:27.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.973+0000 7fd344c2c700 1 --2- 172.21.15.57:0/1446363003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340106490 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd3340098a0 tx=0x7fd334009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e18abd5d348f91 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:27.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.973+0000 7fd33f7fe700 1 -- 172.21.15.57:0/1446363003 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd334004160 con 0x7fd340106070 2023-07-20T18:51:27.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.973+0000 7fd33f7fe700 1 -- 172.21.15.57:0/1446363003 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fd3340042c0 con 0x7fd340106070 2023-07-20T18:51:27.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.973+0000 7fd33f7fe700 1 -- 172.21.15.57:0/1446363003 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd334011ac0 con 0x7fd340106070 2023-07-20T18:51:27.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.973+0000 7fd346e90700 1 -- 172.21.15.57:0/1446363003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 msgr2=0x7fd340106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:27.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.973+0000 7fd346e90700 1 --2- 172.21.15.57:0/1446363003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340106490 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd3340098a0 tx=0x7fd334009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:27.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.973+0000 7fd346e90700 1 -- 172.21.15.57:0/1446363003 shutdown_connections 2023-07-20T18:51:27.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.973+0000 7fd346e90700 1 --2- 172.21.15.57:0/1446363003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340106490 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:27.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.973+0000 7fd346e90700 1 -- 172.21.15.57:0/1446363003 >> 172.21.15.57:0/1446363003 conn(0x7fd340101610 msgr2=0x7fd340103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:27.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.974+0000 7fd346e90700 1 -- 172.21.15.57:0/1446363003 shutdown_connections 2023-07-20T18:51:27.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.974+0000 7fd346e90700 1 -- 172.21.15.57:0/1446363003 wait complete. 2023-07-20T18:51:27.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.975+0000 7fd346e90700 1 Processor -- start 2023-07-20T18:51:27.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.975+0000 7fd346e90700 1 -- start start 2023-07-20T18:51:27.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.976+0000 7fd346e90700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:27.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.976+0000 7fd346e90700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34019a270 con 0x7fd340106070 2023-07-20T18:51:27.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.976+0000 7fd344c2c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:27.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.976+0000 7fd344c2c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50966/0 (socket says 172.21.15.57:50966) 2023-07-20T18:51:27.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.976+0000 7fd344c2c700 1 -- 172.21.15.57:0/3595120335 learned_addr learned my addr 172.21.15.57:0/3595120335 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:27.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.976+0000 7fd344c2c700 1 -- 172.21.15.57:0/3595120335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd334009550 con 0x7fd340106070 2023-07-20T18:51:27.749 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.977+0000 7fd344c2c700 1 --2- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340199d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd334000c00 tx=0x7fd334011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:27.749 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.977+0000 7fd33dffb700 1 -- 172.21.15.57:0/3595120335 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd334003aa0 con 0x7fd340106070 2023-07-20T18:51:27.749 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.977+0000 7fd346e90700 1 -- 172.21.15.57:0/3595120335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34019a470 con 0x7fd340106070 2023-07-20T18:51:27.749 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.977+0000 7fd33dffb700 1 -- 172.21.15.57:0/3595120335 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fd334003c00 con 0x7fd340106070 2023-07-20T18:51:27.749 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.977+0000 7fd33dffb700 1 -- 172.21.15.57:0/3595120335 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd3340248a0 con 0x7fd340106070 2023-07-20T18:51:27.749 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.977+0000 7fd346e90700 1 -- 172.21.15.57:0/3595120335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34019a890 con 0x7fd340106070 2023-07-20T18:51:27.749 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.978+0000 7fd346e90700 1 -- 172.21.15.57:0/3595120335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34010a4a0 con 0x7fd340106070 2023-07-20T18:51:27.749 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.979+0000 7fd33dffb700 1 -- 172.21.15.57:0/3595120335 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fd33402d3f0 con 0x7fd340106070 2023-07-20T18:51:27.750 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.979+0000 7fd33dffb700 1 --2- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fd3300387d0 0x7fd33003ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:27.750 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.979+0000 7fd33dffb700 1 -- 172.21.15.57:0/3595120335 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd33404c660 con 0x7fd340106070 2023-07-20T18:51:27.750 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.979+0000 7fd33ffff700 1 -- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fd3300387d0 msgr2=0x7fd33003ac90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/195678005 2023-07-20T18:51:27.750 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.979+0000 7fd33ffff700 1 --2- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fd3300387d0 0x7fd33003ac90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T18:51:27.750 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:26.984+0000 7fd33dffb700 1 -- 172.21.15.57:0/3595120335 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd334010740 con 0x7fd340106070 2023-07-20T18:51:27.750 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.180+0000 7fd33ffff700 1 -- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fd3300387d0 msgr2=0x7fd33003ac90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/195678005 2023-07-20T18:51:27.750 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.180+0000 7fd33ffff700 1 --2- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fd3300387d0 0x7fd33003ac90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T18:51:27.751 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.250+0000 7fd346e90700 1 -- 172.21.15.57:0/3595120335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd34019da00 con 0x7fd340106070 2023-07-20T18:51:27.751 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.250+0000 7fd33dffb700 1 -- 172.21.15.57:0/3595120335 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fd33400b3e0 con 0x7fd340106070 2023-07-20T18:51:27.751 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.254+0000 7fd346e90700 1 -- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fd3300387d0 msgr2=0x7fd33003ac90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T18:51:27.751 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.254+0000 7fd346e90700 1 --2- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fd3300387d0 0x7fd33003ac90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:27.751 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.254+0000 7fd346e90700 1 -- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 msgr2=0x7fd340199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:27.751 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.254+0000 7fd346e90700 1 --2- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340199d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd334000c00 tx=0x7fd334011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:27.751 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.254+0000 7fd346e90700 1 -- 172.21.15.57:0/3595120335 shutdown_connections 2023-07-20T18:51:27.752 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.254+0000 7fd346e90700 1 --2- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fd3300387d0 0x7fd33003ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:27.752 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.254+0000 7fd346e90700 1 --2- 172.21.15.57:0/3595120335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340106070 0x7fd340199d30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:27.752 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.254+0000 7fd346e90700 1 -- 172.21.15.57:0/3595120335 >> 172.21.15.57:0/3595120335 conn(0x7fd340101610 msgr2=0x7fd340102380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:27.752 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.254+0000 7fd346e90700 1 -- 172.21.15.57:0/3595120335 shutdown_connections 2023-07-20T18:51:27.752 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:27.254+0000 7fd346e90700 1 -- 172.21.15.57:0/3595120335 wait complete. 2023-07-20T18:51:27.753 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for the mgr to restart... 2023-07-20T18:51:27.753 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mgr epoch 5... 2023-07-20T18:51:27.942 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:27 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:27.465+0000 7fe9f476e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T18:51:27.943 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:27 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:27.594+0000 7fe9f476e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T18:51:28.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:27 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3595120335' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T18:51:28.722 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:28 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:28.191+0000 7fe9f476e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T18:51:29.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:28 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:28.959+0000 7fe9f476e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T18:51:29.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:29 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:29.169+0000 7fe9f476e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T18:51:30.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:30 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:30.146+0000 7fe9f476e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T18:51:30.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:30 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:30.268+0000 7fe9f476e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T18:51:30.903 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:30 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:30.495+0000 7fe9f476e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T18:51:31.186 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:30 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:30.867+0000 7fe9f476e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T18:51:31.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:31 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:31.150+0000 7fe9f476e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T18:51:31.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:31 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:31.346+0000 7fe9f476e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T18:51:32.222 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:31 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:31.695+0000 7fe9f476e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T18:51:32.972 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:32 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:32.649+0000 7fe9f476e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T18:51:32.972 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:32 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:32.826+0000 7fe9f476e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T18:51:33.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:33 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:33.077+0000 7fe9f476e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T18:51:33.472 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:33 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:33.227+0000 7fe9f476e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T18:51:33.972 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:33 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:33.514+0000 7fe9f476e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T18:51:34.972 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:34.612+0000 7fe9f476e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T18:51:34.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: Active manager daemon x restarted 2023-07-20T18:51:34.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: Activating manager daemon x 2023-07-20T18:51:34.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: osdmap e2: 0 total, 0 up, 0 in 2023-07-20T18:51:34.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: mgrmap e6: x(active, starting, since 0.00897908s) 2023-07-20T18:51:34.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:51:34.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T18:51:34.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T18:51:34.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T18:51:34.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T18:51:34.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: Manager daemon x is now available 2023-07-20T18:51:34.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:34.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:34.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:34 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:51:35.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:35 smithi057 ceph-mon[126340]: Found migration_current of "None". Setting to last migration. 2023-07-20T18:51:35.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:35 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:51:35.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:35 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:51:35.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:35 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T18:51:35.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:35 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T18:51:35.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:35 smithi057 ceph-mon[126340]: mgrmap e7: x(active, since 1.00744s) 2023-07-20T18:51:36.069 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2023-07-20T18:51:36.070 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-20T18:51:36.070 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T18:51:36.070 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2023-07-20T18:51:36.070 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.255+0000 7fec9f2bf700 1 Processor -- start 2023-07-20T18:51:36.071 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.255+0000 7fec9f2bf700 1 -- start start 2023-07-20T18:51:36.071 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.256+0000 7fec9f2bf700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:36.071 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.256+0000 7fec9f2bf700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec98106a60 con 0x7fec98106070 2023-07-20T18:51:36.071 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.256+0000 7fec9d05b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:36.072 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.256+0000 7fec9d05b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50970/0 (socket says 172.21.15.57:50970) 2023-07-20T18:51:36.072 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.256+0000 7fec9d05b700 1 -- 172.21.15.57:0/3041077871 learned_addr learned my addr 172.21.15.57:0/3041077871 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:36.072 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.257+0000 7fec9d05b700 1 -- 172.21.15.57:0/3041077871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec98107270 con 0x7fec98106070 2023-07-20T18:51:36.073 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.257+0000 7fec9d05b700 1 --2- 172.21.15.57:0/3041077871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98106490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fec940098a0 tx=0x7fec94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5182cd0f7ebf8f30 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:36.073 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.257+0000 7fec8ffff700 1 -- 172.21.15.57:0/3041077871 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec94004160 con 0x7fec98106070 2023-07-20T18:51:36.073 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.257+0000 7fec8ffff700 1 -- 172.21.15.57:0/3041077871 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fec940042c0 con 0x7fec98106070 2023-07-20T18:51:36.073 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.257+0000 7fec8ffff700 1 -- 172.21.15.57:0/3041077871 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec94011ac0 con 0x7fec98106070 2023-07-20T18:51:36.074 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.258+0000 7fec9f2bf700 1 -- 172.21.15.57:0/3041077871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 msgr2=0x7fec98106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:36.074 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.258+0000 7fec9f2bf700 1 --2- 172.21.15.57:0/3041077871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98106490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fec940098a0 tx=0x7fec94009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:36.074 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.258+0000 7fec9f2bf700 1 -- 172.21.15.57:0/3041077871 shutdown_connections 2023-07-20T18:51:36.074 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.258+0000 7fec9f2bf700 1 --2- 172.21.15.57:0/3041077871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98106490 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:36.075 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.258+0000 7fec9f2bf700 1 -- 172.21.15.57:0/3041077871 >> 172.21.15.57:0/3041077871 conn(0x7fec98101610 msgr2=0x7fec98103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:36.075 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.258+0000 7fec9f2bf700 1 -- 172.21.15.57:0/3041077871 shutdown_connections 2023-07-20T18:51:36.075 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.258+0000 7fec9f2bf700 1 -- 172.21.15.57:0/3041077871 wait complete. 2023-07-20T18:51:36.075 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.259+0000 7fec9f2bf700 1 Processor -- start 2023-07-20T18:51:36.076 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.260+0000 7fec9f2bf700 1 -- start start 2023-07-20T18:51:36.076 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.260+0000 7fec9f2bf700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:36.076 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.260+0000 7fec9f2bf700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9819a270 con 0x7fec98106070 2023-07-20T18:51:36.077 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.260+0000 7fec9d05b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:36.077 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.260+0000 7fec9d05b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50984/0 (socket says 172.21.15.57:50984) 2023-07-20T18:51:36.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.260+0000 7fec9d05b700 1 -- 172.21.15.57:0/1105351511 learned_addr learned my addr 172.21.15.57:0/1105351511 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:36.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.261+0000 7fec9d05b700 1 -- 172.21.15.57:0/1105351511 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec94009550 con 0x7fec98106070 2023-07-20T18:51:36.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.261+0000 7fec9d05b700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98199d30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fec94003f70 tx=0x7fec94011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:36.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.261+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec940039f0 con 0x7fec98106070 2023-07-20T18:51:36.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.261+0000 7fec9f2bf700 1 -- 172.21.15.57:0/1105351511 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9819a470 con 0x7fec98106070 2023-07-20T18:51:36.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.261+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fec94003b50 con 0x7fec98106070 2023-07-20T18:51:36.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.261+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec9401a690 con 0x7fec98106070 2023-07-20T18:51:36.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.262+0000 7fec9f2bf700 1 -- 172.21.15.57:0/1105351511 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9819a890 con 0x7fec98106070 2023-07-20T18:51:36.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.263+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fec94024430 con 0x7fec98106070 2023-07-20T18:51:36.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.263+0000 7fec8e7fc700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 0x7fec8403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:36.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.264+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 --> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec8403b350 con 0x7fec84038780 2023-07-20T18:51:36.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.264+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fec9404cc10 con 0x7fec98106070 2023-07-20T18:51:36.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.264+0000 7fec9c85a700 1 -- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 msgr2=0x7fec8403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/195678005 2023-07-20T18:51:36.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.264+0000 7fec9c85a700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 0x7fec8403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T18:51:36.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.465+0000 7fec9c85a700 1 -- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 msgr2=0x7fec8403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/195678005 2023-07-20T18:51:36.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.465+0000 7fec9c85a700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 0x7fec8403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T18:51:36.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.866+0000 7fec9c85a700 1 -- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 msgr2=0x7fec8403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/195678005 2023-07-20T18:51:36.083 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:28.866+0000 7fec9c85a700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 0x7fec8403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T18:51:36.083 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:29.667+0000 7fec9c85a700 1 -- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 msgr2=0x7fec8403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/195678005 2023-07-20T18:51:36.083 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:29.667+0000 7fec9c85a700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 0x7fec8403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T18:51:36.084 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:31.268+0000 7fec9c85a700 1 -- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 msgr2=0x7fec8403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/195678005 2023-07-20T18:51:36.084 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:31.268+0000 7fec9c85a700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 0x7fec8403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T18:51:36.084 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:34.472+0000 7fec9c85a700 1 -- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 msgr2=0x7fec8403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/195678005 2023-07-20T18:51:36.084 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:34.472+0000 7fec9c85a700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 0x7fec8403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-07-20T18:51:36.085 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:34.622+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7fec940193c0 con 0x7fec98106070 2023-07-20T18:51:36.085 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:34.622+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 msgr2=0x7fec8403ac40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T18:51:36.085 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:34.622+0000 7fec8e7fc700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 0x7fec8403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:36.086 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.620+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fec9402cc10 con 0x7fec98106070 2023-07-20T18:51:36.086 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.620+0000 7fec8e7fc700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fec8403bfc0 0x7fec8403e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:36.086 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.620+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 --> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec8403b350 con 0x7fec8403bfc0 2023-07-20T18:51:36.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.620+0000 7fec9c85a700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fec8403bfc0 0x7fec8403e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:36.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.621+0000 7fec9c85a700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fec8403bfc0 0x7fec8403e3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fec88003a10 tx=0x7fec88009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:36.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.622+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 <== mgr.14118 v2:172.21.15.57:6800/2188156324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fec8403b350 con 0x7fec8403bfc0 2023-07-20T18:51:36.087 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec9f2bf700 1 -- 172.21.15.57:0/1105351511 --> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fec9819afe0 con 0x7fec8403bfc0 2023-07-20T18:51:36.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec8e7fc700 1 -- 172.21.15.57:0/1105351511 <== mgr.14118 v2:172.21.15.57:6800/2188156324 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fec9819afe0 con 0x7fec8403bfc0 2023-07-20T18:51:36.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec9f2bf700 1 -- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fec8403bfc0 msgr2=0x7fec8403e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:36.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec9f2bf700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fec8403bfc0 0x7fec8403e3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fec88003a10 tx=0x7fec88009290 comp rx=0 tx=0).stop 2023-07-20T18:51:36.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec9f2bf700 1 -- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 msgr2=0x7fec98199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:36.088 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec9f2bf700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98199d30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fec94003f70 tx=0x7fec94011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:36.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec9f2bf700 1 -- 172.21.15.57:0/1105351511 shutdown_connections 2023-07-20T18:51:36.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec9f2bf700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec98106070 0x7fec98199d30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:36.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec9f2bf700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fec8403bfc0 0x7fec8403e3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:36.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec9f2bf700 1 --2- 172.21.15.57:0/1105351511 >> [v2:172.21.15.57:6800/195678005,v1:172.21.15.57:6801/195678005] conn(0x7fec84038780 0x7fec8403ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:36.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.634+0000 7fec9f2bf700 1 -- 172.21.15.57:0/1105351511 >> 172.21.15.57:0/1105351511 conn(0x7fec98101610 msgr2=0x7fec98102380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:36.089 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.635+0000 7fec9f2bf700 1 -- 172.21.15.57:0/1105351511 shutdown_connections 2023-07-20T18:51:36.090 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:35.635+0000 7fec9f2bf700 1 -- 172.21.15.57:0/1105351511 wait complete. 2023-07-20T18:51:36.090 INFO:teuthology.orchestra.run.smithi057.stdout:mgr epoch 5 is available 2023-07-20T18:51:36.090 INFO:teuthology.orchestra.run.smithi057.stdout:Setting orchestrator backend to cephadm... 2023-07-20T18:51:36.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:36 smithi057 ceph-mon[126340]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T18:51:36.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:36 smithi057 ceph-mon[126340]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T18:51:37.362 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.578+0000 7fdca8e73700 1 Processor -- start 2023-07-20T18:51:37.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.578+0000 7fdca8e73700 1 -- start start 2023-07-20T18:51:37.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.579+0000 7fdca8e73700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca407a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:37.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.579+0000 7fdca8e73700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca407a890 con 0x7fdca407be60 2023-07-20T18:51:37.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.579+0000 7fdca259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca407a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:37.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.579+0000 7fdca259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca407a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60422/0 (socket says 172.21.15.57:60422) 2023-07-20T18:51:37.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.579+0000 7fdca259c700 1 -- 172.21.15.57:0/1679867189 learned_addr learned my addr 172.21.15.57:0/1679867189 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:37.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.579+0000 7fdca259c700 1 -- 172.21.15.57:0/1679867189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca407a9d0 con 0x7fdca407be60 2023-07-20T18:51:37.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.580+0000 7fdca259c700 1 --2- 172.21.15.57:0/1679867189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca407a2c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc940098a0 tx=0x7fdc94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b2d3945ffd045 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:37.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.580+0000 7fdca159a700 1 -- 172.21.15.57:0/1679867189 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc94004160 con 0x7fdca407be60 2023-07-20T18:51:37.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.580+0000 7fdca159a700 1 -- 172.21.15.57:0/1679867189 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fdc940042c0 con 0x7fdca407be60 2023-07-20T18:51:37.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.580+0000 7fdca159a700 1 -- 172.21.15.57:0/1679867189 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc94011ac0 con 0x7fdca407be60 2023-07-20T18:51:37.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.580+0000 7fdca8e73700 1 -- 172.21.15.57:0/1679867189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 msgr2=0x7fdca407a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:37.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.580+0000 7fdca8e73700 1 --2- 172.21.15.57:0/1679867189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca407a2c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc940098a0 tx=0x7fdc94009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:37.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.581+0000 7fdca8e73700 1 -- 172.21.15.57:0/1679867189 shutdown_connections 2023-07-20T18:51:37.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.581+0000 7fdca8e73700 1 --2- 172.21.15.57:0/1679867189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca407a2c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:37.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.581+0000 7fdca8e73700 1 -- 172.21.15.57:0/1679867189 >> 172.21.15.57:0/1679867189 conn(0x7fdca4101610 msgr2=0x7fdca4103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:37.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.581+0000 7fdca8e73700 1 -- 172.21.15.57:0/1679867189 shutdown_connections 2023-07-20T18:51:37.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.581+0000 7fdca8e73700 1 -- 172.21.15.57:0/1679867189 wait complete. 2023-07-20T18:51:37.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.582+0000 7fdca8e73700 1 Processor -- start 2023-07-20T18:51:37.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.582+0000 7fdca8e73700 1 -- start start 2023-07-20T18:51:37.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.582+0000 7fdca8e73700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca419bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:37.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.582+0000 7fdca8e73700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca419c400 con 0x7fdca407be60 2023-07-20T18:51:37.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.582+0000 7fdca259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca419bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:37.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.582+0000 7fdca259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca419bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60436/0 (socket says 172.21.15.57:60436) 2023-07-20T18:51:37.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.582+0000 7fdca259c700 1 -- 172.21.15.57:0/1801825027 learned_addr learned my addr 172.21.15.57:0/1801825027 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:37.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.583+0000 7fdca259c700 1 -- 172.21.15.57:0/1801825027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc94009550 con 0x7fdca407be60 2023-07-20T18:51:37.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.583+0000 7fdca259c700 1 --2- 172.21.15.57:0/1801825027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca419bec0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009130 tx=0x7fdc94011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:37.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.583+0000 7fdc937fe700 1 -- 172.21.15.57:0/1801825027 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc94003aa0 con 0x7fdca407be60 2023-07-20T18:51:37.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.583+0000 7fdc937fe700 1 -- 172.21.15.57:0/1801825027 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fdc94003c00 con 0x7fdca407be60 2023-07-20T18:51:37.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.583+0000 7fdca8e73700 1 -- 172.21.15.57:0/1801825027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca419c600 con 0x7fdca407be60 2023-07-20T18:51:37.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.583+0000 7fdc937fe700 1 -- 172.21.15.57:0/1801825027 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc94024800 con 0x7fdca407be60 2023-07-20T18:51:37.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.584+0000 7fdca8e73700 1 -- 172.21.15.57:0/1801825027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca419ca20 con 0x7fdca407be60 2023-07-20T18:51:37.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.585+0000 7fdc937fe700 1 -- 172.21.15.57:0/1801825027 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fdc9401abd0 con 0x7fdca407be60 2023-07-20T18:51:37.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.585+0000 7fdca8e73700 1 -- 172.21.15.57:0/1801825027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca4061da0 con 0x7fdca407be60 2023-07-20T18:51:37.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.585+0000 7fdc937fe700 1 --2- 172.21.15.57:0/1801825027 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdc8c0386f0 0x7fdc8c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:37.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.585+0000 7fdc937fe700 1 -- 172.21.15.57:0/1801825027 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdc9404c5f0 con 0x7fdca407be60 2023-07-20T18:51:37.369 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.586+0000 7fdca1d9b700 1 --2- 172.21.15.57:0/1801825027 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdc8c0386f0 0x7fdc8c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:37.369 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.587+0000 7fdca1d9b700 1 --2- 172.21.15.57:0/1801825027 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdc8c0386f0 0x7fdc8c03abb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc98006fd0 tx=0x7fdc98006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:37.369 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.590+0000 7fdc937fe700 1 -- 172.21.15.57:0/1801825027 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdc94018760 con 0x7fdca407be60 2023-07-20T18:51:37.369 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.678+0000 7fdc937fe700 1 -- 172.21.15.57:0/1801825027 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fdc9401a430 con 0x7fdca407be60 2023-07-20T18:51:37.369 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.813+0000 7fdca8e73700 1 -- 172.21.15.57:0/1801825027 --> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fdca4102ac0 con 0x7fdc8c0386f0 2023-07-20T18:51:37.369 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.820+0000 7fdc937fe700 1 -- 172.21.15.57:0/1801825027 <== mgr.14118 v2:172.21.15.57:6800/2188156324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdca4102ac0 con 0x7fdc8c0386f0 2023-07-20T18:51:37.369 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.823+0000 7fdca8e73700 1 -- 172.21.15.57:0/1801825027 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdc8c0386f0 msgr2=0x7fdc8c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:37.369 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.823+0000 7fdca8e73700 1 --2- 172.21.15.57:0/1801825027 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdc8c0386f0 0x7fdc8c03abb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc98006fd0 tx=0x7fdc98006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:37.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.823+0000 7fdca8e73700 1 -- 172.21.15.57:0/1801825027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 msgr2=0x7fdca419bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:37.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.823+0000 7fdca8e73700 1 --2- 172.21.15.57:0/1801825027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca419bec0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009130 tx=0x7fdc94011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:37.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.823+0000 7fdca8e73700 1 -- 172.21.15.57:0/1801825027 shutdown_connections 2023-07-20T18:51:37.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.823+0000 7fdca8e73700 1 --2- 172.21.15.57:0/1801825027 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdc8c0386f0 0x7fdc8c03abb0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:37.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.823+0000 7fdca8e73700 1 --2- 172.21.15.57:0/1801825027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca407be60 0x7fdca419bec0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:37.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.823+0000 7fdca8e73700 1 -- 172.21.15.57:0/1801825027 >> 172.21.15.57:0/1801825027 conn(0x7fdca4101610 msgr2=0x7fdca41023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:37.370 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.823+0000 7fdca8e73700 1 -- 172.21.15.57:0/1801825027 shutdown_connections 2023-07-20T18:51:37.371 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:36.823+0000 7fdca8e73700 1 -- 172.21.15.57:0/1801825027 wait complete. 2023-07-20T18:51:37.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:37 smithi057 ceph-mon[126340]: mgrmap e8: x(active, since 2s) 2023-07-20T18:51:37.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:37 smithi057 ceph-mon[126340]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:51:37.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:37 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:37.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:37 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:51:37.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:37 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:37.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:37 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:37.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:37 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:51:38.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-20T18:51:38.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.986+0000 7f9f9e321700 1 Processor -- start 2023-07-20T18:51:38.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.986+0000 7f9f9e321700 1 -- start start 2023-07-20T18:51:38.741 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.986+0000 7f9f9e321700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9807a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:38.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.986+0000 7f9f9e321700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9807a8d0 con 0x7f9f9807bea0 2023-07-20T18:51:38.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.986+0000 7f9f9d31f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9807a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:38.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.987+0000 7f9f9d31f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9807a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60438/0 (socket says 172.21.15.57:60438) 2023-07-20T18:51:38.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.987+0000 7f9f9d31f700 1 -- 172.21.15.57:0/2260605976 learned_addr learned my addr 172.21.15.57:0/2260605976 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:38.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.987+0000 7f9f9d31f700 1 -- 172.21.15.57:0/2260605976 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9807aa10 con 0x7f9f9807bea0 2023-07-20T18:51:38.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.987+0000 7f9f9d31f700 1 --2- 172.21.15.57:0/2260605976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9807a300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f940098a0 tx=0x7f9f94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7636b0982862a42 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:38.742 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.987+0000 7f9f8ffff700 1 -- 172.21.15.57:0/2260605976 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f94004160 con 0x7f9f9807bea0 2023-07-20T18:51:38.743 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.988+0000 7f9f8ffff700 1 -- 172.21.15.57:0/2260605976 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f9f940042c0 con 0x7f9f9807bea0 2023-07-20T18:51:38.743 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.988+0000 7f9f8ffff700 1 -- 172.21.15.57:0/2260605976 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f94011ac0 con 0x7f9f9807bea0 2023-07-20T18:51:38.743 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.988+0000 7f9f9e321700 1 -- 172.21.15.57:0/2260605976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 msgr2=0x7f9f9807a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:38.743 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.988+0000 7f9f9e321700 1 --2- 172.21.15.57:0/2260605976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9807a300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f940098a0 tx=0x7f9f94009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:38.743 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.988+0000 7f9f9e321700 1 -- 172.21.15.57:0/2260605976 shutdown_connections 2023-07-20T18:51:38.743 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.988+0000 7f9f9e321700 1 --2- 172.21.15.57:0/2260605976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9807a300 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:38.743 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.988+0000 7f9f9e321700 1 -- 172.21.15.57:0/2260605976 >> 172.21.15.57:0/2260605976 conn(0x7f9f98101430 msgr2=0x7f9f98103870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:38.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.988+0000 7f9f9e321700 1 -- 172.21.15.57:0/2260605976 shutdown_connections 2023-07-20T18:51:38.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.988+0000 7f9f9e321700 1 -- 172.21.15.57:0/2260605976 wait complete. 2023-07-20T18:51:38.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.989+0000 7f9f9e321700 1 Processor -- start 2023-07-20T18:51:38.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.989+0000 7f9f9e321700 1 -- start start 2023-07-20T18:51:38.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.990+0000 7f9f9e321700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:38.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.990+0000 7f9f9e321700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9819c280 con 0x7f9f9807bea0 2023-07-20T18:51:38.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.990+0000 7f9f9d31f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:38.744 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.990+0000 7f9f9d31f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9819bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60444/0 (socket says 172.21.15.57:60444) 2023-07-20T18:51:38.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.990+0000 7f9f9d31f700 1 -- 172.21.15.57:0/1545011308 learned_addr learned my addr 172.21.15.57:0/1545011308 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:38.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.990+0000 7f9f9d31f700 1 -- 172.21.15.57:0/1545011308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f94009550 con 0x7f9f9807bea0 2023-07-20T18:51:38.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.991+0000 7f9f9d31f700 1 --2- 172.21.15.57:0/1545011308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9819bd40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f94003f70 tx=0x7f9f94011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:38.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.991+0000 7f9f8e7fc700 1 -- 172.21.15.57:0/1545011308 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f940039f0 con 0x7f9f9807bea0 2023-07-20T18:51:38.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.991+0000 7f9f8e7fc700 1 -- 172.21.15.57:0/1545011308 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f9f94003b50 con 0x7f9f9807bea0 2023-07-20T18:51:38.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.991+0000 7f9f9e321700 1 -- 172.21.15.57:0/1545011308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9819c480 con 0x7f9f9807bea0 2023-07-20T18:51:38.745 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.991+0000 7f9f8e7fc700 1 -- 172.21.15.57:0/1545011308 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f9401a690 con 0x7f9f9807bea0 2023-07-20T18:51:38.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.991+0000 7f9f9e321700 1 -- 172.21.15.57:0/1545011308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9819c8a0 con 0x7f9f9807bea0 2023-07-20T18:51:38.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.992+0000 7f9f9e321700 1 -- 172.21.15.57:0/1545011308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f98061da0 con 0x7f9f9807bea0 2023-07-20T18:51:38.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.992+0000 7f9f8e7fc700 1 -- 172.21.15.57:0/1545011308 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f9f94024430 con 0x7f9f9807bea0 2023-07-20T18:51:38.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.993+0000 7f9f8e7fc700 1 --2- 172.21.15.57:0/1545011308 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f9f840387d0 0x7f9f8403ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:38.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.993+0000 7f9f8e7fc700 1 -- 172.21.15.57:0/1545011308 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9f9404c940 con 0x7f9f9807bea0 2023-07-20T18:51:38.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.993+0000 7f9f9cb1e700 1 --2- 172.21.15.57:0/1545011308 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f9f840387d0 0x7f9f8403ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:38.746 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.993+0000 7f9f9cb1e700 1 --2- 172.21.15.57:0/1545011308 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f9f840387d0 0x7f9f8403ac90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9f88006fd0 tx=0x7f9f88006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:38.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:37.998+0000 7f9f8e7fc700 1 -- 172.21.15.57:0/1545011308 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9f940189a0 con 0x7f9f9807bea0 2023-07-20T18:51:38.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.207+0000 7f9f9e321700 1 -- 172.21.15.57:0/1545011308 --> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f98102910 con 0x7f9f840387d0 2023-07-20T18:51:38.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.208+0000 7f9f8e7fc700 1 -- 172.21.15.57:0/1545011308 <== mgr.14118 v2:172.21.15.57:6800/2188156324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f9f98102910 con 0x7f9f840387d0 2023-07-20T18:51:38.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.210+0000 7f9f9e321700 1 -- 172.21.15.57:0/1545011308 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f9f840387d0 msgr2=0x7f9f8403ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:38.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.210+0000 7f9f9e321700 1 --2- 172.21.15.57:0/1545011308 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f9f840387d0 0x7f9f8403ac90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9f88006fd0 tx=0x7f9f88006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:38.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.210+0000 7f9f9e321700 1 -- 172.21.15.57:0/1545011308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 msgr2=0x7f9f9819bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:38.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.210+0000 7f9f9e321700 1 --2- 172.21.15.57:0/1545011308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9819bd40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f94003f70 tx=0x7f9f94011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:38.747 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.210+0000 7f9f9e321700 1 -- 172.21.15.57:0/1545011308 shutdown_connections 2023-07-20T18:51:38.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.210+0000 7f9f9e321700 1 --2- 172.21.15.57:0/1545011308 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f9f840387d0 0x7f9f8403ac90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:38.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.210+0000 7f9f9e321700 1 --2- 172.21.15.57:0/1545011308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f9807bea0 0x7f9f9819bd40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:38.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.210+0000 7f9f9e321700 1 -- 172.21.15.57:0/1545011308 >> 172.21.15.57:0/1545011308 conn(0x7f9f98101430 msgr2=0x7f9f98102200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:38.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.210+0000 7f9f9e321700 1 -- 172.21.15.57:0/1545011308 shutdown_connections 2023-07-20T18:51:38.748 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:38.210+0000 7f9f9e321700 1 -- 172.21.15.57:0/1545011308 wait complete. 2023-07-20T18:51:38.748 INFO:teuthology.orchestra.run.smithi057.stdout:Generating ssh key... 2023-07-20T18:51:39.022 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:38 smithi057 ceph-mon[126340]: [20/Jul/2023:18:51:37] ENGINE Bus STARTING 2023-07-20T18:51:39.022 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:38 smithi057 ceph-mon[126340]: [20/Jul/2023:18:51:37] ENGINE Serving on http://172.21.15.57:8765 2023-07-20T18:51:39.022 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:38 smithi057 ceph-mon[126340]: [20/Jul/2023:18:51:37] ENGINE Serving on https://172.21.15.57:7150 2023-07-20T18:51:39.022 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:38 smithi057 ceph-mon[126340]: [20/Jul/2023:18:51:37] ENGINE Bus STARTED 2023-07-20T18:51:40.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-mon[126340]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:51:40.223 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: Generating public/private rsa key pair. 2023-07-20T18:51:40.223 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: Your identification has been saved in /tmp/tmpvb11thws/key. 2023-07-20T18:51:40.223 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: Your public key has been saved in /tmp/tmpvb11thws/key.pub. 2023-07-20T18:51:40.223 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: The key fingerprint is: 2023-07-20T18:51:40.224 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: SHA256:3JrYckk/F/EXg50q3T866mTgZ2JOFSnaR48CUk2Jy5Y ceph-38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:51:40.224 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: The key's randomart image is: 2023-07-20T18:51:40.224 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: +---[RSA 3072]----+ 2023-07-20T18:51:40.224 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: | .+.. | 2023-07-20T18:51:40.225 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: | .. o . o .| 2023-07-20T18:51:40.225 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: | ...o. +.. = | 2023-07-20T18:51:40.225 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: | .E+.o =oo o| 2023-07-20T18:51:40.225 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: | ..S+.=.+...| 2023-07-20T18:51:40.225 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: | +.== .. ..| 2023-07-20T18:51:40.226 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: | o *=o=. ..| 2023-07-20T18:51:40.226 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: | o+ *o .. .| 2023-07-20T18:51:40.226 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: | ..o... | 2023-07-20T18:51:40.226 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: +----[SHA256]-----+ 2023-07-20T18:51:40.321 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.250+0000 7fdd4d1f8700 1 Processor -- start 2023-07-20T18:51:40.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.251+0000 7fdd4d1f8700 1 -- start start 2023-07-20T18:51:40.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.251+0000 7fdd4d1f8700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4807a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:40.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.251+0000 7fdd4d1f8700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4807a8f0 con 0x7fdd4807bec0 2023-07-20T18:51:40.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.251+0000 7fdd46d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4807a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:40.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.251+0000 7fdd46d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4807a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60446/0 (socket says 172.21.15.57:60446) 2023-07-20T18:51:40.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.251+0000 7fdd46d9d700 1 -- 172.21.15.57:0/4169841456 learned_addr learned my addr 172.21.15.57:0/4169841456 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:40.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.252+0000 7fdd46d9d700 1 -- 172.21.15.57:0/4169841456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4807aa30 con 0x7fdd4807bec0 2023-07-20T18:51:40.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.252+0000 7fdd46d9d700 1 --2- 172.21.15.57:0/4169841456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4807a320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdd380098a0 tx=0x7fdd38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e13432fed17f7c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:40.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.252+0000 7fdd45d9b700 1 -- 172.21.15.57:0/4169841456 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd38004160 con 0x7fdd4807bec0 2023-07-20T18:51:40.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.252+0000 7fdd45d9b700 1 -- 172.21.15.57:0/4169841456 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fdd380042c0 con 0x7fdd4807bec0 2023-07-20T18:51:40.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.252+0000 7fdd45d9b700 1 -- 172.21.15.57:0/4169841456 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd38011ac0 con 0x7fdd4807bec0 2023-07-20T18:51:40.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.253+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/4169841456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 msgr2=0x7fdd4807a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:40.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.253+0000 7fdd4d1f8700 1 --2- 172.21.15.57:0/4169841456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4807a320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdd380098a0 tx=0x7fdd38009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:40.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.253+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/4169841456 shutdown_connections 2023-07-20T18:51:40.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.253+0000 7fdd4d1f8700 1 --2- 172.21.15.57:0/4169841456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4807a320 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:40.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.253+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/4169841456 >> 172.21.15.57:0/4169841456 conn(0x7fdd48101610 msgr2=0x7fdd48103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:40.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.253+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/4169841456 shutdown_connections 2023-07-20T18:51:40.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.253+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/4169841456 wait complete. 2023-07-20T18:51:40.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.254+0000 7fdd4d1f8700 1 Processor -- start 2023-07-20T18:51:40.326 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.254+0000 7fdd4d1f8700 1 -- start start 2023-07-20T18:51:40.326 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.254+0000 7fdd4d1f8700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4819bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:40.326 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.254+0000 7fdd4d1f8700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4819c440 con 0x7fdd4807bec0 2023-07-20T18:51:40.326 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.255+0000 7fdd46d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4819bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:40.326 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.255+0000 7fdd46d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4819bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60458/0 (socket says 172.21.15.57:60458) 2023-07-20T18:51:40.327 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.255+0000 7fdd46d9d700 1 -- 172.21.15.57:0/1216872743 learned_addr learned my addr 172.21.15.57:0/1216872743 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:40.327 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.255+0000 7fdd46d9d700 1 -- 172.21.15.57:0/1216872743 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd38009550 con 0x7fdd4807bec0 2023-07-20T18:51:40.327 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.255+0000 7fdd46d9d700 1 --2- 172.21.15.57:0/1216872743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4819bf00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdd38000c00 tx=0x7fdd38011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:40.327 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.255+0000 7fdd2ffff700 1 -- 172.21.15.57:0/1216872743 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd38003aa0 con 0x7fdd4807bec0 2023-07-20T18:51:40.327 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.256+0000 7fdd2ffff700 1 -- 172.21.15.57:0/1216872743 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fdd38003c00 con 0x7fdd4807bec0 2023-07-20T18:51:40.328 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.256+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/1216872743 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4819c640 con 0x7fdd4807bec0 2023-07-20T18:51:40.328 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.256+0000 7fdd2ffff700 1 -- 172.21.15.57:0/1216872743 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd380248a0 con 0x7fdd4807bec0 2023-07-20T18:51:40.328 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.256+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/1216872743 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4819ca60 con 0x7fdd4807bec0 2023-07-20T18:51:40.329 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.257+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/1216872743 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd48061da0 con 0x7fdd4807bec0 2023-07-20T18:51:40.329 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.257+0000 7fdd2ffff700 1 -- 172.21.15.57:0/1216872743 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fdd3801a430 con 0x7fdd4807bec0 2023-07-20T18:51:40.329 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.257+0000 7fdd2ffff700 1 --2- 172.21.15.57:0/1216872743 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdd30038820 0x7fdd3003ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:40.329 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.257+0000 7fdd2ffff700 1 -- 172.21.15.57:0/1216872743 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdd3804c800 con 0x7fdd4807bec0 2023-07-20T18:51:40.330 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.258+0000 7fdd4659c700 1 --2- 172.21.15.57:0/1216872743 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdd30038820 0x7fdd3003ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:40.330 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.258+0000 7fdd4659c700 1 --2- 172.21.15.57:0/1216872743 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdd30038820 0x7fdd3003ace0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c006fd0 tx=0x7fdd3c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:40.330 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.262+0000 7fdd2ffff700 1 -- 172.21.15.57:0/1216872743 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdd38018530 con 0x7fdd4807bec0 2023-07-20T18:51:40.331 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.469+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/1216872743 --> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd48102af0 con 0x7fdd30038820 2023-07-20T18:51:40.331 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.808+0000 7fdd2ffff700 1 -- 172.21.15.57:0/1216872743 <== mgr.14118 v2:172.21.15.57:6800/2188156324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdd48102af0 con 0x7fdd30038820 2023-07-20T18:51:40.331 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.810+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/1216872743 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdd30038820 msgr2=0x7fdd3003ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:40.331 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.810+0000 7fdd4d1f8700 1 --2- 172.21.15.57:0/1216872743 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdd30038820 0x7fdd3003ace0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c006fd0 tx=0x7fdd3c006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:40.332 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.810+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/1216872743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 msgr2=0x7fdd4819bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:40.332 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.810+0000 7fdd4d1f8700 1 --2- 172.21.15.57:0/1216872743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4819bf00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdd38000c00 tx=0x7fdd38011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:40.332 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.810+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/1216872743 shutdown_connections 2023-07-20T18:51:40.332 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.810+0000 7fdd4d1f8700 1 --2- 172.21.15.57:0/1216872743 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fdd30038820 0x7fdd3003ace0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:40.333 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.810+0000 7fdd4d1f8700 1 --2- 172.21.15.57:0/1216872743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4807bec0 0x7fdd4819bf00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:40.333 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.810+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/1216872743 >> 172.21.15.57:0/1216872743 conn(0x7fdd48101610 msgr2=0x7fdd481023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:40.333 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.810+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/1216872743 shutdown_connections 2023-07-20T18:51:40.333 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:39.810+0000 7fdd4d1f8700 1 -- 172.21.15.57:0/1216872743 wait complete. 2023-07-20T18:51:40.972 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:40 smithi057 ceph-mon[126340]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:51:40.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:40 smithi057 ceph-mon[126340]: Generating ssh key... 2023-07-20T18:51:40.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:40 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:40.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:40 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:41.603 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6TRNby6luT5sHlY1YKnUvVFG2ORB3FJ1J14fQejW9LpsMw4qjVkPA0pJjKEWfjqDUTI8jZb6M9/M59LcuelRsPOAhogtK5JGkXiVFsI2EgxIh59CCX5UwHVXjs6F2yrogEJv5paJr8iCAvv/XINlv2vKnfw5Jpvlhd+MFJ7FC+tiZfrRK2M1EBv2MKdqsA8fZNoR2R6mrAqlksdz2DCTwCCUAL8WKsAIcUcZO9Dvmm5oJn7+W9aT5avy/btj+y2fEX/+wcnqbAUJ3kZJfkQIOWKxDMxVjH1QxZrDhiY7Vk5Frq61ADKWc9uEDtQkq96YlQr9Yr0CZcv3FDwLz80GXnn0e7ZVm87dD25Pzrd825YBh3Vptfmv0GaIBwrBsSTvG0SjOwTlDsVWht9NdtkSDA+lj698a2BKKR8Y0xVVsoYNh7Y8S2tTjuiNN3RCaMmmg9T965hcRcAZARpY5N9JLfVprRh1bVkM4gwQBHbIcnJ+0YRQXljsN/zIxVHC5d+U= ceph-38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:51:41.604 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.836+0000 7f820300b700 1 Processor -- start 2023-07-20T18:51:41.604 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.836+0000 7f820300b700 1 -- start start 2023-07-20T18:51:41.604 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.836+0000 7f820300b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc07a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:41.604 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.836+0000 7f820300b700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc07a8d0 con 0x7f81fc07bea0 2023-07-20T18:51:41.604 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.836+0000 7f8200da7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc07a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:41.605 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.837+0000 7f8200da7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc07a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60460/0 (socket says 172.21.15.57:60460) 2023-07-20T18:51:41.605 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.837+0000 7f8200da7700 1 -- 172.21.15.57:0/3441191330 learned_addr learned my addr 172.21.15.57:0/3441191330 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:41.605 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.837+0000 7f8200da7700 1 -- 172.21.15.57:0/3441191330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc07aa10 con 0x7f81fc07bea0 2023-07-20T18:51:41.605 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.837+0000 7f8200da7700 1 --2- 172.21.15.57:0/3441191330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc07a300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f81f00098a0 tx=0x7f81f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359fed6a56b8da97 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:41.606 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.837+0000 7f81fb7fe700 1 -- 172.21.15.57:0/3441191330 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81f0004160 con 0x7f81fc07bea0 2023-07-20T18:51:41.606 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.837+0000 7f81fb7fe700 1 -- 172.21.15.57:0/3441191330 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f81f00042c0 con 0x7f81fc07bea0 2023-07-20T18:51:41.606 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.837+0000 7f81fb7fe700 1 -- 172.21.15.57:0/3441191330 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81f0011ac0 con 0x7f81fc07bea0 2023-07-20T18:51:41.606 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.838+0000 7f820300b700 1 -- 172.21.15.57:0/3441191330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 msgr2=0x7f81fc07a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:41.606 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.838+0000 7f820300b700 1 --2- 172.21.15.57:0/3441191330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc07a300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f81f00098a0 tx=0x7f81f0009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:41.606 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.838+0000 7f820300b700 1 -- 172.21.15.57:0/3441191330 shutdown_connections 2023-07-20T18:51:41.607 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.838+0000 7f820300b700 1 --2- 172.21.15.57:0/3441191330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc07a300 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:41.607 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.838+0000 7f820300b700 1 -- 172.21.15.57:0/3441191330 >> 172.21.15.57:0/3441191330 conn(0x7f81fc101610 msgr2=0x7f81fc103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:41.607 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.838+0000 7f820300b700 1 -- 172.21.15.57:0/3441191330 shutdown_connections 2023-07-20T18:51:41.607 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.838+0000 7f820300b700 1 -- 172.21.15.57:0/3441191330 wait complete. 2023-07-20T18:51:41.607 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.839+0000 7f820300b700 1 Processor -- start 2023-07-20T18:51:41.607 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.839+0000 7f820300b700 1 -- start start 2023-07-20T18:51:41.607 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.840+0000 7f820300b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc19bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:41.608 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.840+0000 7f820300b700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc19c470 con 0x7f81fc07bea0 2023-07-20T18:51:41.608 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.840+0000 7f8200da7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc19bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:41.608 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.840+0000 7f8200da7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc19bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60470/0 (socket says 172.21.15.57:60470) 2023-07-20T18:51:41.608 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.840+0000 7f8200da7700 1 -- 172.21.15.57:0/2947612315 learned_addr learned my addr 172.21.15.57:0/2947612315 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:41.608 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.840+0000 7f8200da7700 1 -- 172.21.15.57:0/2947612315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f0009550 con 0x7f81fc07bea0 2023-07-20T18:51:41.608 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.840+0000 7f8200da7700 1 --2- 172.21.15.57:0/2947612315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc19bf30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f81f0000c00 tx=0x7f81f0011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:41.608 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.841+0000 7f81f9ffb700 1 -- 172.21.15.57:0/2947612315 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81f0003aa0 con 0x7f81fc07bea0 2023-07-20T18:51:41.609 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.841+0000 7f81f9ffb700 1 -- 172.21.15.57:0/2947612315 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f81f0003c00 con 0x7f81fc07bea0 2023-07-20T18:51:41.609 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.841+0000 7f820300b700 1 -- 172.21.15.57:0/2947612315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81fc19c670 con 0x7f81fc07bea0 2023-07-20T18:51:41.609 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.841+0000 7f81f9ffb700 1 -- 172.21.15.57:0/2947612315 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81f00248a0 con 0x7f81fc07bea0 2023-07-20T18:51:41.609 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.841+0000 7f820300b700 1 -- 172.21.15.57:0/2947612315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81fc19ca90 con 0x7f81fc07bea0 2023-07-20T18:51:41.609 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.842+0000 7f81f9ffb700 1 -- 172.21.15.57:0/2947612315 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f81f001ac60 con 0x7f81fc07bea0 2023-07-20T18:51:41.609 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.842+0000 7f820300b700 1 -- 172.21.15.57:0/2947612315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81fc061da0 con 0x7f81fc07bea0 2023-07-20T18:51:41.609 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.842+0000 7f81f9ffb700 1 --2- 172.21.15.57:0/2947612315 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f81ec038870 0x7f81ec03ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:41.610 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.843+0000 7f81f9ffb700 1 -- 172.21.15.57:0/2947612315 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f81f004c790 con 0x7f81fc07bea0 2023-07-20T18:51:41.610 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.843+0000 7f81fbfff700 1 --2- 172.21.15.57:0/2947612315 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f81ec038870 0x7f81ec03ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:41.610 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.843+0000 7f81fbfff700 1 --2- 172.21.15.57:0/2947612315 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f81ec038870 0x7f81ec03ad30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f81e8006fd0 tx=0x7f81e8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:41.610 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:40.847+0000 7f81f9ffb700 1 -- 172.21.15.57:0/2947612315 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f81f001a430 con 0x7f81fc07bea0 2023-07-20T18:51:41.610 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.056+0000 7f820300b700 1 -- 172.21.15.57:0/2947612315 --> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f81fc102b10 con 0x7f81ec038870 2023-07-20T18:51:41.610 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.057+0000 7f81f9ffb700 1 -- 172.21.15.57:0/2947612315 <== mgr.14118 v2:172.21.15.57:6800/2188156324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f81fc102b10 con 0x7f81ec038870 2023-07-20T18:51:41.610 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.059+0000 7f820300b700 1 -- 172.21.15.57:0/2947612315 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f81ec038870 msgr2=0x7f81ec03ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:41.611 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.059+0000 7f820300b700 1 --2- 172.21.15.57:0/2947612315 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f81ec038870 0x7f81ec03ad30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f81e8006fd0 tx=0x7f81e8006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:41.611 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.059+0000 7f820300b700 1 -- 172.21.15.57:0/2947612315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 msgr2=0x7f81fc19bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:41.611 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.059+0000 7f820300b700 1 --2- 172.21.15.57:0/2947612315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc19bf30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f81f0000c00 tx=0x7f81f0011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:41.612 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.059+0000 7f820300b700 1 -- 172.21.15.57:0/2947612315 shutdown_connections 2023-07-20T18:51:41.612 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.059+0000 7f820300b700 1 --2- 172.21.15.57:0/2947612315 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f81ec038870 0x7f81ec03ad30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:41.612 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.059+0000 7f820300b700 1 --2- 172.21.15.57:0/2947612315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc07bea0 0x7f81fc19bf30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:41.612 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.059+0000 7f820300b700 1 -- 172.21.15.57:0/2947612315 >> 172.21.15.57:0/2947612315 conn(0x7f81fc101610 msgr2=0x7f81fc102400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:41.612 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.059+0000 7f820300b700 1 -- 172.21.15.57:0/2947612315 shutdown_connections 2023-07-20T18:51:41.612 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:41.059+0000 7f820300b700 1 -- 172.21.15.57:0/2947612315 wait complete. 2023-07-20T18:51:41.612 INFO:teuthology.orchestra.run.smithi057.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-20T18:51:41.613 INFO:teuthology.orchestra.run.smithi057.stdout:Adding key to root@localhost authorized_keys... 2023-07-20T18:51:41.613 INFO:teuthology.orchestra.run.smithi057.stdout:Adding host smithi057... 2023-07-20T18:51:42.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:42 smithi057 ceph-mon[126340]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:51:43.966 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:43 smithi057 ceph-mon[126340]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi057", "addr": "172.21.15.57", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:51:45.064 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:44 smithi057 ceph-mon[126340]: Deploying cephadm binary to smithi057 2023-07-20T18:51:46.091 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout Added host 'smithi057' with addr '172.21.15.57' 2023-07-20T18:51:46.092 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.129+0000 7fa1768bd700 1 Processor -- start 2023-07-20T18:51:46.092 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.129+0000 7fa1768bd700 1 -- start start 2023-07-20T18:51:46.092 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.129+0000 7fa1768bd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa170106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:46.092 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.129+0000 7fa1768bd700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa170106a60 con 0x7fa170106070 2023-07-20T18:51:46.092 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.130+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa170106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:46.092 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.130+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa170106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60478/0 (socket says 172.21.15.57:60478) 2023-07-20T18:51:46.092 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.130+0000 7fa16ffff700 1 -- 172.21.15.57:0/2407511976 learned_addr learned my addr 172.21.15.57:0/2407511976 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:46.093 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.130+0000 7fa16ffff700 1 -- 172.21.15.57:0/2407511976 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa170107270 con 0x7fa170106070 2023-07-20T18:51:46.093 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.130+0000 7fa16ffff700 1 --2- 172.21.15.57:0/2407511976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa170106490 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa1600098a0 tx=0x7fa160009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88856b813c966dde server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:46.093 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.130+0000 7fa16effd700 1 -- 172.21.15.57:0/2407511976 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa160004160 con 0x7fa170106070 2023-07-20T18:51:46.093 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.131+0000 7fa16effd700 1 -- 172.21.15.57:0/2407511976 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fa1600042c0 con 0x7fa170106070 2023-07-20T18:51:46.093 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.131+0000 7fa16effd700 1 -- 172.21.15.57:0/2407511976 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa160011ac0 con 0x7fa170106070 2023-07-20T18:51:46.093 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.131+0000 7fa1768bd700 1 -- 172.21.15.57:0/2407511976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 msgr2=0x7fa170106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:46.093 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.131+0000 7fa1768bd700 1 --2- 172.21.15.57:0/2407511976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa170106490 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa1600098a0 tx=0x7fa160009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:46.094 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.131+0000 7fa1768bd700 1 -- 172.21.15.57:0/2407511976 shutdown_connections 2023-07-20T18:51:46.094 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.131+0000 7fa1768bd700 1 --2- 172.21.15.57:0/2407511976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa170106490 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:46.094 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.131+0000 7fa1768bd700 1 -- 172.21.15.57:0/2407511976 >> 172.21.15.57:0/2407511976 conn(0x7fa170101610 msgr2=0x7fa170103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:46.094 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.131+0000 7fa1768bd700 1 -- 172.21.15.57:0/2407511976 shutdown_connections 2023-07-20T18:51:46.094 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.131+0000 7fa1768bd700 1 -- 172.21.15.57:0/2407511976 wait complete. 2023-07-20T18:51:46.094 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.132+0000 7fa1768bd700 1 Processor -- start 2023-07-20T18:51:46.094 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.132+0000 7fa1768bd700 1 -- start start 2023-07-20T18:51:46.095 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.133+0000 7fa1768bd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa17019bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:46.095 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.133+0000 7fa1768bd700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17019c450 con 0x7fa170106070 2023-07-20T18:51:46.095 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.133+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa17019bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:46.095 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.133+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa17019bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60484/0 (socket says 172.21.15.57:60484) 2023-07-20T18:51:46.095 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.133+0000 7fa16ffff700 1 -- 172.21.15.57:0/2773388836 learned_addr learned my addr 172.21.15.57:0/2773388836 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:46.095 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.133+0000 7fa16ffff700 1 -- 172.21.15.57:0/2773388836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa160009550 con 0x7fa170106070 2023-07-20T18:51:46.095 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.134+0000 7fa16ffff700 1 --2- 172.21.15.57:0/2773388836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa17019bf10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa160000c00 tx=0x7fa160011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:46.096 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.134+0000 7fa16d7fa700 1 -- 172.21.15.57:0/2773388836 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa160003aa0 con 0x7fa170106070 2023-07-20T18:51:46.096 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.134+0000 7fa16d7fa700 1 -- 172.21.15.57:0/2773388836 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7fa160003c00 con 0x7fa170106070 2023-07-20T18:51:46.096 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.134+0000 7fa16d7fa700 1 -- 172.21.15.57:0/2773388836 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1600248a0 con 0x7fa170106070 2023-07-20T18:51:46.096 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.134+0000 7fa1768bd700 1 -- 172.21.15.57:0/2773388836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17019c650 con 0x7fa170106070 2023-07-20T18:51:46.096 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.134+0000 7fa1768bd700 1 -- 172.21.15.57:0/2773388836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17019ca70 con 0x7fa170106070 2023-07-20T18:51:46.096 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.135+0000 7fa1768bd700 1 -- 172.21.15.57:0/2773388836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa170061da0 con 0x7fa170106070 2023-07-20T18:51:46.096 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.135+0000 7fa16d7fa700 1 -- 172.21.15.57:0/2773388836 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fa16001ac60 con 0x7fa170106070 2023-07-20T18:51:46.096 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.136+0000 7fa16d7fa700 1 --2- 172.21.15.57:0/2773388836 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fa158038820 0x7fa15803ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:46.097 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.136+0000 7fa16d7fa700 1 -- 172.21.15.57:0/2773388836 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa16004c740 con 0x7fa170106070 2023-07-20T18:51:46.097 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.136+0000 7fa16f7fe700 1 --2- 172.21.15.57:0/2773388836 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fa158038820 0x7fa15803ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:46.097 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.136+0000 7fa16f7fe700 1 --2- 172.21.15.57:0/2773388836 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fa158038820 0x7fa15803ace0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa164006fd0 tx=0x7fa164006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:46.097 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.140+0000 7fa16d7fa700 1 -- 172.21.15.57:0/2773388836 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa160018530 con 0x7fa170106070 2023-07-20T18:51:46.097 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:42.351+0000 7fa1768bd700 1 -- 172.21.15.57:0/2773388836 --> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi057", "addr": "172.21.15.57", "target": ["mon-mgr", ""]}) v1 -- 0x7fa170102a50 con 0x7fa158038820 2023-07-20T18:51:46.097 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.567+0000 7fa16d7fa700 1 -- 172.21.15.57:0/2773388836 <== mgr.14118 v2:172.21.15.57:6800/2188156324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fa170102a50 con 0x7fa158038820 2023-07-20T18:51:46.097 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.569+0000 7fa1768bd700 1 -- 172.21.15.57:0/2773388836 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fa158038820 msgr2=0x7fa15803ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.569+0000 7fa1768bd700 1 --2- 172.21.15.57:0/2773388836 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fa158038820 0x7fa15803ace0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa164006fd0 tx=0x7fa164006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.569+0000 7fa1768bd700 1 -- 172.21.15.57:0/2773388836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 msgr2=0x7fa17019bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.569+0000 7fa1768bd700 1 --2- 172.21.15.57:0/2773388836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa17019bf10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa160000c00 tx=0x7fa160011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.569+0000 7fa1768bd700 1 -- 172.21.15.57:0/2773388836 shutdown_connections 2023-07-20T18:51:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.569+0000 7fa1768bd700 1 --2- 172.21.15.57:0/2773388836 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7fa158038820 0x7fa15803ace0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.569+0000 7fa1768bd700 1 --2- 172.21.15.57:0/2773388836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170106070 0x7fa17019bf10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.569+0000 7fa1768bd700 1 -- 172.21.15.57:0/2773388836 >> 172.21.15.57:0/2773388836 conn(0x7fa170101610 msgr2=0x7fa170102340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.570+0000 7fa1768bd700 1 -- 172.21.15.57:0/2773388836 shutdown_connections 2023-07-20T18:51:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:45.570+0000 7fa1768bd700 1 -- 172.21.15.57:0/2773388836 wait complete. 2023-07-20T18:51:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:Deploying unmanaged mon service... 2023-07-20T18:51:46.722 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:46 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:46.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:46 smithi057 ceph-mon[126340]: Added host smithi057 2023-07-20T18:51:46.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:46 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:51:47.502 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-20T18:51:47.502 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.666+0000 7f65c68d8700 1 Processor -- start 2023-07-20T18:51:47.503 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.666+0000 7f65c68d8700 1 -- start start 2023-07-20T18:51:47.503 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.666+0000 7f65c68d8700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c007a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:47.503 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.666+0000 7f65c68d8700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c007a8f0 con 0x7f65c007bec0 2023-07-20T18:51:47.503 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.667+0000 7f65bffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c007a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:47.503 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.667+0000 7f65bffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c007a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48306/0 (socket says 172.21.15.57:48306) 2023-07-20T18:51:47.503 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.667+0000 7f65bffff700 1 -- 172.21.15.57:0/2429903047 learned_addr learned my addr 172.21.15.57:0/2429903047 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:47.503 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.667+0000 7f65bffff700 1 -- 172.21.15.57:0/2429903047 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c007aa30 con 0x7f65c007bec0 2023-07-20T18:51:47.504 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.667+0000 7f65bffff700 1 --2- 172.21.15.57:0/2429903047 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c007a320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f65b00098a0 tx=0x7f65b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befb992cf41f1f19 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:47.504 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.667+0000 7f65beffd700 1 -- 172.21.15.57:0/2429903047 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b0004160 con 0x7f65c007bec0 2023-07-20T18:51:47.504 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.668+0000 7f65beffd700 1 -- 172.21.15.57:0/2429903047 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f65b00042c0 con 0x7f65c007bec0 2023-07-20T18:51:47.504 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.668+0000 7f65beffd700 1 -- 172.21.15.57:0/2429903047 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b0011ac0 con 0x7f65c007bec0 2023-07-20T18:51:47.504 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.668+0000 7f65c68d8700 1 -- 172.21.15.57:0/2429903047 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 msgr2=0x7f65c007a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:47.504 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.668+0000 7f65c68d8700 1 --2- 172.21.15.57:0/2429903047 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c007a320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f65b00098a0 tx=0x7f65b0009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:47.505 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.668+0000 7f65c68d8700 1 -- 172.21.15.57:0/2429903047 shutdown_connections 2023-07-20T18:51:47.505 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.668+0000 7f65c68d8700 1 --2- 172.21.15.57:0/2429903047 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c007a320 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:47.505 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.668+0000 7f65c68d8700 1 -- 172.21.15.57:0/2429903047 >> 172.21.15.57:0/2429903047 conn(0x7f65c0101610 msgr2=0x7f65c0103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:47.505 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.668+0000 7f65c68d8700 1 -- 172.21.15.57:0/2429903047 shutdown_connections 2023-07-20T18:51:47.505 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.668+0000 7f65c68d8700 1 -- 172.21.15.57:0/2429903047 wait complete. 2023-07-20T18:51:47.505 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.670+0000 7f65c68d8700 1 Processor -- start 2023-07-20T18:51:47.509 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.670+0000 7f65c68d8700 1 -- start start 2023-07-20T18:51:47.510 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.671+0000 7f65c68d8700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c019bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:47.510 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.671+0000 7f65c68d8700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c019c440 con 0x7f65c007bec0 2023-07-20T18:51:47.510 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.671+0000 7f65bffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c019bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:47.510 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.671+0000 7f65bffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c019bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48322/0 (socket says 172.21.15.57:48322) 2023-07-20T18:51:47.510 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.671+0000 7f65bffff700 1 -- 172.21.15.57:0/852654390 learned_addr learned my addr 172.21.15.57:0/852654390 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:47.510 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.671+0000 7f65bffff700 1 -- 172.21.15.57:0/852654390 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b0009550 con 0x7f65c007bec0 2023-07-20T18:51:47.510 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.671+0000 7f65bffff700 1 --2- 172.21.15.57:0/852654390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c019bf00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65b0000c00 tx=0x7f65b0011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:47.511 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.672+0000 7f65bd7fa700 1 -- 172.21.15.57:0/852654390 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b0003aa0 con 0x7f65c007bec0 2023-07-20T18:51:47.511 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.672+0000 7f65bd7fa700 1 -- 172.21.15.57:0/852654390 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f65b0003c00 con 0x7f65c007bec0 2023-07-20T18:51:47.511 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.672+0000 7f65c68d8700 1 -- 172.21.15.57:0/852654390 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c019c640 con 0x7f65c007bec0 2023-07-20T18:51:47.511 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.672+0000 7f65bd7fa700 1 -- 172.21.15.57:0/852654390 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b00248a0 con 0x7f65c007bec0 2023-07-20T18:51:47.511 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.672+0000 7f65c68d8700 1 -- 172.21.15.57:0/852654390 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c019ca60 con 0x7f65c007bec0 2023-07-20T18:51:47.511 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.673+0000 7f65c68d8700 1 -- 172.21.15.57:0/852654390 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c010c480 con 0x7f65c007bec0 2023-07-20T18:51:47.512 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.673+0000 7f65bd7fa700 1 -- 172.21.15.57:0/852654390 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f65b001ac60 con 0x7f65c007bec0 2023-07-20T18:51:47.512 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.674+0000 7f65bd7fa700 1 --2- 172.21.15.57:0/852654390 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f65a8038820 0x7f65a803ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:47.512 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.674+0000 7f65bd7fa700 1 -- 172.21.15.57:0/852654390 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f65b004c800 con 0x7f65c007bec0 2023-07-20T18:51:47.512 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.674+0000 7f65bf7fe700 1 --2- 172.21.15.57:0/852654390 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f65a8038820 0x7f65a803ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:47.512 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.675+0000 7f65bf7fe700 1 --2- 172.21.15.57:0/852654390 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f65a8038820 0x7f65a803ace0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f65b4006fd0 tx=0x7f65b4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:47.512 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.679+0000 7f65bd7fa700 1 -- 172.21.15.57:0/852654390 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f65b0018530 con 0x7f65c007bec0 2023-07-20T18:51:47.513 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.901+0000 7f65c68d8700 1 -- 172.21.15.57:0/852654390 --> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f65c019f8c0 con 0x7f65a8038820 2023-07-20T18:51:47.513 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.906+0000 7f65bd7fa700 1 -- 172.21.15.57:0/852654390 <== mgr.14118 v2:172.21.15.57:6800/2188156324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f65c019f8c0 con 0x7f65a8038820 2023-07-20T18:51:47.513 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.909+0000 7f65c68d8700 1 -- 172.21.15.57:0/852654390 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f65a8038820 msgr2=0x7f65a803ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:47.513 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.909+0000 7f65c68d8700 1 --2- 172.21.15.57:0/852654390 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f65a8038820 0x7f65a803ace0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f65b4006fd0 tx=0x7f65b4006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:47.513 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.909+0000 7f65c68d8700 1 -- 172.21.15.57:0/852654390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 msgr2=0x7f65c019bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:47.513 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.909+0000 7f65c68d8700 1 --2- 172.21.15.57:0/852654390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c019bf00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65b0000c00 tx=0x7f65b0011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:47.514 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.909+0000 7f65c68d8700 1 -- 172.21.15.57:0/852654390 shutdown_connections 2023-07-20T18:51:47.514 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.909+0000 7f65c68d8700 1 --2- 172.21.15.57:0/852654390 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f65a8038820 0x7f65a803ace0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:47.514 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.909+0000 7f65c68d8700 1 --2- 172.21.15.57:0/852654390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c007bec0 0x7f65c019bf00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:47.514 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.909+0000 7f65c68d8700 1 -- 172.21.15.57:0/852654390 >> 172.21.15.57:0/852654390 conn(0x7f65c0101610 msgr2=0x7f65c01023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:47.514 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.909+0000 7f65c68d8700 1 -- 172.21.15.57:0/852654390 shutdown_connections 2023-07-20T18:51:47.515 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:46.909+0000 7f65c68d8700 1 -- 172.21.15.57:0/852654390 wait complete. 2023-07-20T18:51:47.515 INFO:teuthology.orchestra.run.smithi057.stdout:Deploying unmanaged mgr service... 2023-07-20T18:51:48.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:47 smithi057 ceph-mon[126340]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:51:48.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:47 smithi057 ceph-mon[126340]: Saving service mon spec with placement count:5 2023-07-20T18:51:48.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:47 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:48.936 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-20T18:51:48.936 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.234+0000 7f83b1e99700 1 Processor -- start 2023-07-20T18:51:48.936 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.234+0000 7f83b1e99700 1 -- start start 2023-07-20T18:51:48.936 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.235+0000 7f83b1e99700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:48.937 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.235+0000 7f83b1e99700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac106a60 con 0x7f83ac106070 2023-07-20T18:51:48.937 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.235+0000 7f83ab7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:48.937 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.235+0000 7f83ab7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48334/0 (socket says 172.21.15.57:48334) 2023-07-20T18:51:48.937 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.235+0000 7f83ab7fe700 1 -- 172.21.15.57:0/369275755 learned_addr learned my addr 172.21.15.57:0/369275755 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:48.937 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.235+0000 7f83ab7fe700 1 -- 172.21.15.57:0/369275755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac107270 con 0x7f83ac106070 2023-07-20T18:51:48.937 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.236+0000 7f83ab7fe700 1 --2- 172.21.15.57:0/369275755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac106490 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f839c0098a0 tx=0x7f839c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c374fb9d5f45b4 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:48.937 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.236+0000 7f83aa7fc700 1 -- 172.21.15.57:0/369275755 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f839c004160 con 0x7f83ac106070 2023-07-20T18:51:48.937 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.236+0000 7f83aa7fc700 1 -- 172.21.15.57:0/369275755 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f839c0042c0 con 0x7f83ac106070 2023-07-20T18:51:48.938 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.236+0000 7f83aa7fc700 1 -- 172.21.15.57:0/369275755 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f839c011ac0 con 0x7f83ac106070 2023-07-20T18:51:48.938 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.236+0000 7f83b1e99700 1 -- 172.21.15.57:0/369275755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 msgr2=0x7f83ac106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:48.938 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.236+0000 7f83b1e99700 1 --2- 172.21.15.57:0/369275755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac106490 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f839c0098a0 tx=0x7f839c009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:48.938 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.237+0000 7f83b1e99700 1 -- 172.21.15.57:0/369275755 shutdown_connections 2023-07-20T18:51:48.938 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.237+0000 7f83b1e99700 1 --2- 172.21.15.57:0/369275755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac106490 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:48.938 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.237+0000 7f83b1e99700 1 -- 172.21.15.57:0/369275755 >> 172.21.15.57:0/369275755 conn(0x7f83ac101610 msgr2=0x7f83ac103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:48.938 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.237+0000 7f83b1e99700 1 -- 172.21.15.57:0/369275755 shutdown_connections 2023-07-20T18:51:48.939 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.237+0000 7f83b1e99700 1 -- 172.21.15.57:0/369275755 wait complete. 2023-07-20T18:51:48.939 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.238+0000 7f83b1e99700 1 Processor -- start 2023-07-20T18:51:48.939 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.238+0000 7f83b1e99700 1 -- start start 2023-07-20T18:51:48.939 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.238+0000 7f83b1e99700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac07bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:48.939 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.238+0000 7f83b1e99700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac07c190 con 0x7f83ac106070 2023-07-20T18:51:48.939 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.239+0000 7f83ab7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac07bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:48.939 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.239+0000 7f83ab7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac07bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48336/0 (socket says 172.21.15.57:48336) 2023-07-20T18:51:48.939 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.239+0000 7f83ab7fe700 1 -- 172.21.15.57:0/2795931852 learned_addr learned my addr 172.21.15.57:0/2795931852 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:48.940 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.239+0000 7f83ab7fe700 1 -- 172.21.15.57:0/2795931852 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c009550 con 0x7f83ac106070 2023-07-20T18:51:48.940 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.239+0000 7f83ab7fe700 1 --2- 172.21.15.57:0/2795931852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac07bc50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f839c000c00 tx=0x7f839c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:48.940 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.239+0000 7f83a8ff9700 1 -- 172.21.15.57:0/2795931852 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f839c003aa0 con 0x7f83ac106070 2023-07-20T18:51:48.940 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.239+0000 7f83a8ff9700 1 -- 172.21.15.57:0/2795931852 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1108+0+0 (secure 0 0 0) 0x7f839c003c00 con 0x7f83ac106070 2023-07-20T18:51:48.940 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.239+0000 7f83b1e99700 1 -- 172.21.15.57:0/2795931852 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac07a2a0 con 0x7f83ac106070 2023-07-20T18:51:48.940 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.240+0000 7f83a8ff9700 1 -- 172.21.15.57:0/2795931852 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f839c0248a0 con 0x7f83ac106070 2023-07-20T18:51:48.940 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.240+0000 7f83b1e99700 1 -- 172.21.15.57:0/2795931852 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac07a6c0 con 0x7f83ac106070 2023-07-20T18:51:48.941 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.241+0000 7f83b1e99700 1 -- 172.21.15.57:0/2795931852 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac10a4a0 con 0x7f83ac106070 2023-07-20T18:51:48.941 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.241+0000 7f83a8ff9700 1 -- 172.21.15.57:0/2795931852 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f839c01ac60 con 0x7f83ac106070 2023-07-20T18:51:48.941 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.241+0000 7f83a8ff9700 1 --2- 172.21.15.57:0/2795931852 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f8394038820 0x7f839403ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:48.941 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.241+0000 7f83a8ff9700 1 -- 172.21.15.57:0/2795931852 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f839c04c800 con 0x7f83ac106070 2023-07-20T18:51:48.941 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.242+0000 7f83aaffd700 1 --2- 172.21.15.57:0/2795931852 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f8394038820 0x7f839403ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:48.941 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.242+0000 7f83aaffd700 1 --2- 172.21.15.57:0/2795931852 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f8394038820 0x7f839403ace0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f83a0006fd0 tx=0x7f83a0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:48.941 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.247+0000 7f83a8ff9700 1 -- 172.21.15.57:0/2795931852 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f839c018530 con 0x7f83ac106070 2023-07-20T18:51:48.941 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.467+0000 7f83b1e99700 1 -- 172.21.15.57:0/2795931852 --> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f83ac102a90 con 0x7f8394038820 2023-07-20T18:51:48.942 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.469+0000 7f83a8ff9700 1 -- 172.21.15.57:0/2795931852 <== mon.0 v2:172.21.15.57:3300/0 7 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f839c018530 con 0x7f83ac106070 2023-07-20T18:51:48.942 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.474+0000 7f83a8ff9700 1 -- 172.21.15.57:0/2795931852 <== mgr.14118 v2:172.21.15.57:6800/2188156324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f83ac102a90 con 0x7f8394038820 2023-07-20T18:51:48.942 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.477+0000 7f83b1e99700 1 -- 172.21.15.57:0/2795931852 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f8394038820 msgr2=0x7f839403ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:48.942 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.477+0000 7f83b1e99700 1 --2- 172.21.15.57:0/2795931852 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f8394038820 0x7f839403ace0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f83a0006fd0 tx=0x7f83a0006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:48.942 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.477+0000 7f83b1e99700 1 -- 172.21.15.57:0/2795931852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 msgr2=0x7f83ac07bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:48.942 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.477+0000 7f83b1e99700 1 --2- 172.21.15.57:0/2795931852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac07bc50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f839c000c00 tx=0x7f839c011b40 comp rx=0 tx=0).stop 2023-07-20T18:51:48.942 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.477+0000 7f83b1e99700 1 -- 172.21.15.57:0/2795931852 shutdown_connections 2023-07-20T18:51:48.943 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.477+0000 7f83b1e99700 1 --2- 172.21.15.57:0/2795931852 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f8394038820 0x7f839403ace0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:48.943 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.477+0000 7f83b1e99700 1 --2- 172.21.15.57:0/2795931852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83ac106070 0x7f83ac07bc50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:48.943 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.477+0000 7f83b1e99700 1 -- 172.21.15.57:0/2795931852 >> 172.21.15.57:0/2795931852 conn(0x7f83ac101610 msgr2=0x7f83ac102380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:48.943 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.477+0000 7f83b1e99700 1 -- 172.21.15.57:0/2795931852 shutdown_connections 2023-07-20T18:51:48.943 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:48.477+0000 7f83b1e99700 1 -- 172.21.15.57:0/2795931852 wait complete. 2023-07-20T18:51:49.722 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:49 smithi057 ceph-mon[126340]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:51:49.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:49 smithi057 ceph-mon[126340]: Saving service mgr spec with placement count:2 2023-07-20T18:51:49.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:49 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:49.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:49 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:49.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:49 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:50.415 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.496+0000 7f1a63e52700 1 Processor -- start 2023-07-20T18:51:50.415 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.496+0000 7f1a63e52700 1 -- start start 2023-07-20T18:51:50.416 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.496+0000 7f1a63e52700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c07a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:50.417 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.496+0000 7f1a63e52700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c07a8f0 con 0x7f1a5c07bec0 2023-07-20T18:51:50.417 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.497+0000 7f1a61bee700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c07a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:50.418 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.497+0000 7f1a61bee700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c07a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48346/0 (socket says 172.21.15.57:48346) 2023-07-20T18:51:50.418 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.497+0000 7f1a61bee700 1 -- 172.21.15.57:0/1302904809 learned_addr learned my addr 172.21.15.57:0/1302904809 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:50.418 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.497+0000 7f1a61bee700 1 -- 172.21.15.57:0/1302904809 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c07aa30 con 0x7f1a5c07bec0 2023-07-20T18:51:50.418 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.498+0000 7f1a61bee700 1 --2- 172.21.15.57:0/1302904809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c07a320 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a580098a0 tx=0x7f1a58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320c877ae0f5de7 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:50.418 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.498+0000 7f1a60bec700 1 -- 172.21.15.57:0/1302904809 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a58004160 con 0x7f1a5c07bec0 2023-07-20T18:51:50.418 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.498+0000 7f1a60bec700 1 -- 172.21.15.57:0/1302904809 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f1a580042c0 con 0x7f1a5c07bec0 2023-07-20T18:51:50.418 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.498+0000 7f1a60bec700 1 -- 172.21.15.57:0/1302904809 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a58011a80 con 0x7f1a5c07bec0 2023-07-20T18:51:50.419 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.498+0000 7f1a63e52700 1 -- 172.21.15.57:0/1302904809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 msgr2=0x7f1a5c07a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:50.419 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.498+0000 7f1a63e52700 1 --2- 172.21.15.57:0/1302904809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c07a320 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a580098a0 tx=0x7f1a58009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:50.419 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.498+0000 7f1a63e52700 1 -- 172.21.15.57:0/1302904809 shutdown_connections 2023-07-20T18:51:50.419 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.498+0000 7f1a63e52700 1 --2- 172.21.15.57:0/1302904809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c07a320 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:50.419 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.498+0000 7f1a63e52700 1 -- 172.21.15.57:0/1302904809 >> 172.21.15.57:0/1302904809 conn(0x7f1a5c101610 msgr2=0x7f1a5c103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:50.419 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.499+0000 7f1a63e52700 1 -- 172.21.15.57:0/1302904809 shutdown_connections 2023-07-20T18:51:50.419 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.499+0000 7f1a63e52700 1 -- 172.21.15.57:0/1302904809 wait complete. 2023-07-20T18:51:50.419 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.500+0000 7f1a63e52700 1 Processor -- start 2023-07-20T18:51:50.420 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.500+0000 7f1a63e52700 1 -- start start 2023-07-20T18:51:50.420 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.500+0000 7f1a63e52700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c19bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:50.420 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.500+0000 7f1a63e52700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c19c400 con 0x7f1a5c07bec0 2023-07-20T18:51:50.420 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.500+0000 7f1a61bee700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c19bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:50.420 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.500+0000 7f1a61bee700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c19bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48360/0 (socket says 172.21.15.57:48360) 2023-07-20T18:51:50.420 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.500+0000 7f1a61bee700 1 -- 172.21.15.57:0/1036574443 learned_addr learned my addr 172.21.15.57:0/1036574443 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:50.420 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.501+0000 7f1a61bee700 1 -- 172.21.15.57:0/1036574443 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a58009550 con 0x7f1a5c07bec0 2023-07-20T18:51:50.421 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.501+0000 7f1a61bee700 1 --2- 172.21.15.57:0/1036574443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c19bec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1a58003f20 tx=0x7f1a58011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:50.421 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.501+0000 7f1a52ffd700 1 -- 172.21.15.57:0/1036574443 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a58003b20 con 0x7f1a5c07bec0 2023-07-20T18:51:50.421 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.501+0000 7f1a63e52700 1 -- 172.21.15.57:0/1036574443 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c19c600 con 0x7f1a5c07bec0 2023-07-20T18:51:50.421 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.501+0000 7f1a52ffd700 1 -- 172.21.15.57:0/1036574443 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f1a5801a910 con 0x7f1a5c07bec0 2023-07-20T18:51:50.421 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.501+0000 7f1a52ffd700 1 -- 172.21.15.57:0/1036574443 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a58024850 con 0x7f1a5c07bec0 2023-07-20T18:51:50.421 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.501+0000 7f1a63e52700 1 -- 172.21.15.57:0/1036574443 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c19ca20 con 0x7f1a5c07bec0 2023-07-20T18:51:50.421 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.502+0000 7f1a63e52700 1 -- 172.21.15.57:0/1036574443 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c10c4d0 con 0x7f1a5c07bec0 2023-07-20T18:51:50.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.502+0000 7f1a52ffd700 1 -- 172.21.15.57:0/1036574443 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f1a5801aa80 con 0x7f1a5c07bec0 2023-07-20T18:51:50.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.503+0000 7f1a52ffd700 1 --2- 172.21.15.57:0/1036574443 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1a480387d0 0x7f1a4803ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:50.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.503+0000 7f1a52ffd700 1 -- 172.21.15.57:0/1036574443 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1a5804ca60 con 0x7f1a5c07bec0 2023-07-20T18:51:50.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.503+0000 7f1a613ed700 1 --2- 172.21.15.57:0/1036574443 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1a480387d0 0x7f1a4803ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:50.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.504+0000 7f1a613ed700 1 --2- 172.21.15.57:0/1036574443 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1a480387d0 0x7f1a4803ac90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c006fd0 tx=0x7f1a4c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:50.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.507+0000 7f1a52ffd700 1 -- 172.21.15.57:0/1036574443 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1a5801ae10 con 0x7f1a5c07bec0 2023-07-20T18:51:50.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.720+0000 7f1a63e52700 1 -- 172.21.15.57:0/1036574443 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f1a5c10c6c0 con 0x7f1a5c07bec0 2023-07-20T18:51:50.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.728+0000 7f1a52ffd700 1 -- 172.21.15.57:0/1036574443 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1a58018b40 con 0x7f1a5c07bec0 2023-07-20T18:51:50.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.731+0000 7f1a63e52700 1 -- 172.21.15.57:0/1036574443 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1a480387d0 msgr2=0x7f1a4803ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:50.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.731+0000 7f1a63e52700 1 --2- 172.21.15.57:0/1036574443 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1a480387d0 0x7f1a4803ac90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c006fd0 tx=0x7f1a4c006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:50.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.731+0000 7f1a63e52700 1 -- 172.21.15.57:0/1036574443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 msgr2=0x7f1a5c19bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:50.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.731+0000 7f1a63e52700 1 --2- 172.21.15.57:0/1036574443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c19bec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1a58003f20 tx=0x7f1a58011b00 comp rx=0 tx=0).stop 2023-07-20T18:51:50.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.731+0000 7f1a63e52700 1 -- 172.21.15.57:0/1036574443 shutdown_connections 2023-07-20T18:51:50.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.731+0000 7f1a63e52700 1 --2- 172.21.15.57:0/1036574443 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1a480387d0 0x7f1a4803ac90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:50.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.731+0000 7f1a63e52700 1 --2- 172.21.15.57:0/1036574443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c07bec0 0x7f1a5c19bec0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:50.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.731+0000 7f1a63e52700 1 -- 172.21.15.57:0/1036574443 >> 172.21.15.57:0/1036574443 conn(0x7f1a5c101610 msgr2=0x7f1a5c102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:50.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.731+0000 7f1a63e52700 1 -- 172.21.15.57:0/1036574443 shutdown_connections 2023-07-20T18:51:50.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:49.731+0000 7f1a63e52700 1 -- 172.21.15.57:0/1036574443 wait complete. 2023-07-20T18:51:51.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:50 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1036574443' entity='client.admin' 2023-07-20T18:51:51.913 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.231+0000 7f9588265700 1 Processor -- start 2023-07-20T18:51:51.913 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.231+0000 7f9588265700 1 -- start start 2023-07-20T18:51:51.913 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.232+0000 7f9588265700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958007a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:51.914 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.232+0000 7f9588265700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958007a890 con 0x7f958007be60 2023-07-20T18:51:51.914 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.232+0000 7f9586001700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958007a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:51.914 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.232+0000 7f9586001700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958007a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48376/0 (socket says 172.21.15.57:48376) 2023-07-20T18:51:51.914 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.232+0000 7f9586001700 1 -- 172.21.15.57:0/2324233634 learned_addr learned my addr 172.21.15.57:0/2324233634 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:51.914 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.233+0000 7f9586001700 1 -- 172.21.15.57:0/2324233634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958007a9d0 con 0x7f958007be60 2023-07-20T18:51:51.915 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.233+0000 7f9586001700 1 --2- 172.21.15.57:0/2324233634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958007a2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f957c0098a0 tx=0x7f957c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8b8f9a08d707f4 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:51.915 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.233+0000 7f9584fff700 1 -- 172.21.15.57:0/2324233634 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f957c004160 con 0x7f958007be60 2023-07-20T18:51:51.915 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.233+0000 7f9584fff700 1 -- 172.21.15.57:0/2324233634 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f957c0042c0 con 0x7f958007be60 2023-07-20T18:51:51.918 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.233+0000 7f9584fff700 1 -- 172.21.15.57:0/2324233634 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f957c011a80 con 0x7f958007be60 2023-07-20T18:51:51.918 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.234+0000 7f9588265700 1 -- 172.21.15.57:0/2324233634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 msgr2=0x7f958007a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:51.919 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.234+0000 7f9588265700 1 --2- 172.21.15.57:0/2324233634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958007a2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f957c0098a0 tx=0x7f957c009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:51.919 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.234+0000 7f9588265700 1 -- 172.21.15.57:0/2324233634 shutdown_connections 2023-07-20T18:51:51.919 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.234+0000 7f9588265700 1 --2- 172.21.15.57:0/2324233634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958007a2c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:51.919 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.234+0000 7f9588265700 1 -- 172.21.15.57:0/2324233634 >> 172.21.15.57:0/2324233634 conn(0x7f9580101610 msgr2=0x7f9580103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:51.919 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.234+0000 7f9588265700 1 -- 172.21.15.57:0/2324233634 shutdown_connections 2023-07-20T18:51:51.919 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.234+0000 7f9588265700 1 -- 172.21.15.57:0/2324233634 wait complete. 2023-07-20T18:51:51.919 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.235+0000 7f9588265700 1 Processor -- start 2023-07-20T18:51:51.920 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.235+0000 7f9588265700 1 -- start start 2023-07-20T18:51:51.920 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.235+0000 7f9588265700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958019be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:51.920 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.235+0000 7f9588265700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958019c3a0 con 0x7f958007be60 2023-07-20T18:51:51.920 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.236+0000 7f9586001700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958019be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:51.920 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.236+0000 7f9586001700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958019be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48380/0 (socket says 172.21.15.57:48380) 2023-07-20T18:51:51.920 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.236+0000 7f9586001700 1 -- 172.21.15.57:0/733303393 learned_addr learned my addr 172.21.15.57:0/733303393 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:51.920 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.236+0000 7f9586001700 1 -- 172.21.15.57:0/733303393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c009550 con 0x7f958007be60 2023-07-20T18:51:51.920 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.236+0000 7f9586001700 1 --2- 172.21.15.57:0/733303393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958019be60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f957c003f20 tx=0x7f957c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:51.921 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.236+0000 7f9576ffd700 1 -- 172.21.15.57:0/733303393 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f957c003b20 con 0x7f958007be60 2023-07-20T18:51:51.921 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.237+0000 7f9576ffd700 1 -- 172.21.15.57:0/733303393 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f957c01a910 con 0x7f958007be60 2023-07-20T18:51:51.921 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.237+0000 7f9588265700 1 -- 172.21.15.57:0/733303393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958019c5a0 con 0x7f958007be60 2023-07-20T18:51:51.921 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.237+0000 7f9576ffd700 1 -- 172.21.15.57:0/733303393 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f957c0248c0 con 0x7f958007be60 2023-07-20T18:51:51.921 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.237+0000 7f9588265700 1 -- 172.21.15.57:0/733303393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958019c9c0 con 0x7f958007be60 2023-07-20T18:51:51.921 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.238+0000 7f9576ffd700 1 -- 172.21.15.57:0/733303393 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f957c01aa80 con 0x7f958007be60 2023-07-20T18:51:51.921 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.238+0000 7f9588265700 1 -- 172.21.15.57:0/733303393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9580061da0 con 0x7f958007be60 2023-07-20T18:51:51.922 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.238+0000 7f9576ffd700 1 --2- 172.21.15.57:0/733303393 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f956c038820 0x7f956c03ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:51.922 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.238+0000 7f9576ffd700 1 -- 172.21.15.57:0/733303393 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f957c04ca40 con 0x7f958007be60 2023-07-20T18:51:51.922 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.238+0000 7f9585800700 1 --2- 172.21.15.57:0/733303393 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f956c038820 0x7f956c03ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:51.922 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.239+0000 7f9585800700 1 --2- 172.21.15.57:0/733303393 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f956c038820 0x7f956c03ace0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9570006fd0 tx=0x7f9570006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:51.922 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.243+0000 7f9576ffd700 1 -- 172.21.15.57:0/733303393 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f957c01adf0 con 0x7f958007be60 2023-07-20T18:51:51.922 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.455+0000 7f9588265700 1 -- 172.21.15.57:0/733303393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f958019f820 con 0x7f958007be60 2023-07-20T18:51:51.922 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.463+0000 7f9576ffd700 1 -- 172.21.15.57:0/733303393 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f957c018b40 con 0x7f958007be60 2023-07-20T18:51:51.923 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.466+0000 7f9588265700 1 -- 172.21.15.57:0/733303393 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f956c038820 msgr2=0x7f956c03ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:51.923 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.466+0000 7f9588265700 1 --2- 172.21.15.57:0/733303393 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f956c038820 0x7f956c03ace0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9570006fd0 tx=0x7f9570006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:51.923 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.466+0000 7f9588265700 1 -- 172.21.15.57:0/733303393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 msgr2=0x7f958019be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:51.923 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.466+0000 7f9588265700 1 --2- 172.21.15.57:0/733303393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958019be60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f957c003f20 tx=0x7f957c011b00 comp rx=0 tx=0).stop 2023-07-20T18:51:51.923 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.466+0000 7f9588265700 1 -- 172.21.15.57:0/733303393 shutdown_connections 2023-07-20T18:51:51.923 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.466+0000 7f9588265700 1 --2- 172.21.15.57:0/733303393 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f956c038820 0x7f956c03ace0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:51.923 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.466+0000 7f9588265700 1 --2- 172.21.15.57:0/733303393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f958007be60 0x7f958019be60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:51.924 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.466+0000 7f9588265700 1 -- 172.21.15.57:0/733303393 >> 172.21.15.57:0/733303393 conn(0x7f9580101610 msgr2=0x7f9580102310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:51.924 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.466+0000 7f9588265700 1 -- 172.21.15.57:0/733303393 shutdown_connections 2023-07-20T18:51:51.924 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:51.466+0000 7f9588265700 1 -- 172.21.15.57:0/733303393 wait complete. 2023-07-20T18:51:51.924 INFO:teuthology.orchestra.run.smithi057.stdout:Enabling the dashboard module... 2023-07-20T18:51:52.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:52 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:52.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:52 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/733303393' entity='client.admin' 2023-07-20T18:51:54.116 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:53 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/511544864' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-20T18:51:54.116 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:53 smithi057 ceph-mon[126340]: from='mgr.14118 172.21.15.57:0/2998338399' entity='mgr.x' 2023-07-20T18:51:54.116 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:53 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: ignoring --setuser ceph since I am not root 2023-07-20T18:51:54.116 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:53 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: ignoring --setgroup ceph since I am not root 2023-07-20T18:51:54.116 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:54 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:54.078+0000 7f042cfd7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T18:51:54.392 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:54 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:54.185+0000 7f042cfd7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T18:51:54.393 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:54 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:54.304+0000 7f042cfd7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T18:51:54.399 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.476+0000 7ff3d412e700 1 Processor -- start 2023-07-20T18:51:54.399 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.476+0000 7ff3d412e700 1 -- start start 2023-07-20T18:51:54.399 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.476+0000 7ff3d412e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:54.399 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.476+0000 7ff3d412e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc079e30 con 0x7ff3cc106b60 2023-07-20T18:51:54.400 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.476+0000 7ff3d1eca700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:54.400 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.476+0000 7ff3d1eca700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc108f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48394/0 (socket says 172.21.15.57:48394) 2023-07-20T18:51:54.400 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.476+0000 7ff3d1eca700 1 -- 172.21.15.57:0/617799724 learned_addr learned my addr 172.21.15.57:0/617799724 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:54.400 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.477+0000 7ff3d1eca700 1 -- 172.21.15.57:0/617799724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc109490 con 0x7ff3cc106b60 2023-07-20T18:51:54.400 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.477+0000 7ff3d1eca700 1 --2- 172.21.15.57:0/617799724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc108f50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80098a0 tx=0x7ff3c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4c9ba0a1ca2749 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:54.400 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.477+0000 7ff3d0ec8700 1 -- 172.21.15.57:0/617799724 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3c8004160 con 0x7ff3cc106b60 2023-07-20T18:51:54.400 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.477+0000 7ff3d0ec8700 1 -- 172.21.15.57:0/617799724 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7ff3c80042c0 con 0x7ff3cc106b60 2023-07-20T18:51:54.401 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.477+0000 7ff3d0ec8700 1 -- 172.21.15.57:0/617799724 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3c8011a80 con 0x7ff3cc106b60 2023-07-20T18:51:54.401 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.478+0000 7ff3d412e700 1 -- 172.21.15.57:0/617799724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 msgr2=0x7ff3cc108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:54.401 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.478+0000 7ff3d412e700 1 --2- 172.21.15.57:0/617799724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc108f50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80098a0 tx=0x7ff3c8009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:54.401 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.478+0000 7ff3d412e700 1 -- 172.21.15.57:0/617799724 shutdown_connections 2023-07-20T18:51:54.401 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.478+0000 7ff3d412e700 1 --2- 172.21.15.57:0/617799724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc108f50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:54.401 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.478+0000 7ff3d412e700 1 -- 172.21.15.57:0/617799724 >> 172.21.15.57:0/617799724 conn(0x7ff3cc100510 msgr2=0x7ff3cc102950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:54.402 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.478+0000 7ff3d412e700 1 -- 172.21.15.57:0/617799724 shutdown_connections 2023-07-20T18:51:54.402 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.478+0000 7ff3d412e700 1 -- 172.21.15.57:0/617799724 wait complete. 2023-07-20T18:51:54.402 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.479+0000 7ff3d412e700 1 Processor -- start 2023-07-20T18:51:54.402 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.479+0000 7ff3d412e700 1 -- start start 2023-07-20T18:51:54.402 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.480+0000 7ff3d412e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc1064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:54.402 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.480+0000 7ff3d412e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc106a00 con 0x7ff3cc106b60 2023-07-20T18:51:54.402 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.480+0000 7ff3d1eca700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc1064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:54.403 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.480+0000 7ff3d1eca700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc1064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48406/0 (socket says 172.21.15.57:48406) 2023-07-20T18:51:54.403 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.480+0000 7ff3d1eca700 1 -- 172.21.15.57:0/511544864 learned_addr learned my addr 172.21.15.57:0/511544864 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:54.403 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.480+0000 7ff3d1eca700 1 -- 172.21.15.57:0/511544864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c8009550 con 0x7ff3cc106b60 2023-07-20T18:51:54.405 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.481+0000 7ff3d1eca700 1 --2- 172.21.15.57:0/511544864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc1064c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8003f20 tx=0x7ff3c8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:54.406 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.481+0000 7ff3c2ffd700 1 -- 172.21.15.57:0/511544864 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3c8003b20 con 0x7ff3cc106b60 2023-07-20T18:51:54.406 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.481+0000 7ff3c2ffd700 1 -- 172.21.15.57:0/511544864 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7ff3c801a910 con 0x7ff3cc106b60 2023-07-20T18:51:54.406 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.481+0000 7ff3d412e700 1 -- 172.21.15.57:0/511544864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc104b10 con 0x7ff3cc106b60 2023-07-20T18:51:54.406 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.481+0000 7ff3c2ffd700 1 -- 172.21.15.57:0/511544864 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3c80248c0 con 0x7ff3cc106b60 2023-07-20T18:51:54.406 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.481+0000 7ff3d412e700 1 -- 172.21.15.57:0/511544864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc104f30 con 0x7ff3cc106b60 2023-07-20T18:51:54.406 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.482+0000 7ff3c2ffd700 1 -- 172.21.15.57:0/511544864 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff3c801a430 con 0x7ff3cc106b60 2023-07-20T18:51:54.407 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.483+0000 7ff3d412e700 1 -- 172.21.15.57:0/511544864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc10a280 con 0x7ff3cc106b60 2023-07-20T18:51:54.407 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.483+0000 7ff3c2ffd700 1 --2- 172.21.15.57:0/511544864 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7ff3b80387d0 0x7ff3b803ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:54.407 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.483+0000 7ff3c2ffd700 1 -- 172.21.15.57:0/511544864 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff3c804cb50 con 0x7ff3cc106b60 2023-07-20T18:51:54.407 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.483+0000 7ff3d16c9700 1 --2- 172.21.15.57:0/511544864 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7ff3b80387d0 0x7ff3b803ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:54.407 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.484+0000 7ff3d16c9700 1 --2- 172.21.15.57:0/511544864 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7ff3b80387d0 0x7ff3b803ac90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006fd0 tx=0x7ff3bc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:54.407 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.488+0000 7ff3c2ffd700 1 -- 172.21.15.57:0/511544864 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff3c801adf0 con 0x7ff3cc106b60 2023-07-20T18:51:54.408 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:52.774+0000 7ff3d412e700 1 -- 172.21.15.57:0/511544864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff3cc10a4a0 con 0x7ff3cc106b60 2023-07-20T18:51:54.408 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.769+0000 7ff3c2ffd700 1 -- 172.21.15.57:0/511544864 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7ff3c802d770 con 0x7ff3cc106b60 2023-07-20T18:51:54.408 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.771+0000 7ff3c2ffd700 1 -- 172.21.15.57:0/511544864 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff3c802a2a0 con 0x7ff3cc106b60 2023-07-20T18:51:54.408 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.774+0000 7ff3d412e700 1 -- 172.21.15.57:0/511544864 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7ff3b80387d0 msgr2=0x7ff3b803ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:54.408 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.774+0000 7ff3d412e700 1 --2- 172.21.15.57:0/511544864 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7ff3b80387d0 0x7ff3b803ac90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc006fd0 tx=0x7ff3bc006e40 comp rx=0 tx=0).stop 2023-07-20T18:51:54.408 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.774+0000 7ff3d412e700 1 -- 172.21.15.57:0/511544864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 msgr2=0x7ff3cc1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.774+0000 7ff3d412e700 1 --2- 172.21.15.57:0/511544864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc1064c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8003f20 tx=0x7ff3c8011b00 comp rx=0 tx=0).stop 2023-07-20T18:51:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.774+0000 7ff3d412e700 1 -- 172.21.15.57:0/511544864 shutdown_connections 2023-07-20T18:51:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.774+0000 7ff3d412e700 1 --2- 172.21.15.57:0/511544864 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7ff3b80387d0 0x7ff3b803ac90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.774+0000 7ff3d412e700 1 --2- 172.21.15.57:0/511544864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3cc106b60 0x7ff3cc1064c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.774+0000 7ff3d412e700 1 -- 172.21.15.57:0/511544864 >> 172.21.15.57:0/511544864 conn(0x7ff3cc100510 msgr2=0x7ff3cc101230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.775+0000 7ff3d412e700 1 -- 172.21.15.57:0/511544864 shutdown_connections 2023-07-20T18:51:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:53.775+0000 7ff3d412e700 1 -- 172.21.15.57:0/511544864 wait complete. 2023-07-20T18:51:54.720 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:54 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:54.441+0000 7f042cfd7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T18:51:54.720 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:54 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:54.682+0000 7f042cfd7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T18:51:54.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:54 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/511544864' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-20T18:51:54.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:54 smithi057 ceph-mon[126340]: mgrmap e9: x(active, since 19s) 2023-07-20T18:51:55.445 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:55.097+0000 7f042cfd7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T18:51:55.445 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:55.228+0000 7f042cfd7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T18:51:55.714 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:55.407+0000 7f042cfd7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T18:51:55.714 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:55.535+0000 7f042cfd7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T18:51:55.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:51:55 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1590438538' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T18:51:56.115 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2023-07-20T18:51:56.115 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-20T18:51:56.115 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T18:51:56.115 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T18:51:56.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T18:51:56.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2023-07-20T18:51:56.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.374+0000 7f08c1c60700 1 Processor -- start 2023-07-20T18:51:56.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.374+0000 7f08c1c60700 1 -- start start 2023-07-20T18:51:56.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.375+0000 7f08c1c60700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc07a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:56.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.375+0000 7f08c1c60700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc07a890 con 0x7f08bc07be60 2023-07-20T18:51:56.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.375+0000 7f08bb7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc07a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:56.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.375+0000 7f08bb7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc07a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33020/0 (socket says 172.21.15.57:33020) 2023-07-20T18:51:56.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.375+0000 7f08bb7fe700 1 -- 172.21.15.57:0/1909192091 learned_addr learned my addr 172.21.15.57:0/1909192091 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:56.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.376+0000 7f08bb7fe700 1 -- 172.21.15.57:0/1909192091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc07a9d0 con 0x7f08bc07be60 2023-07-20T18:51:56.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.376+0000 7f08bb7fe700 1 --2- 172.21.15.57:0/1909192091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc07a2c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0098a0 tx=0x7f08ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4bc5f232346c7b server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:56.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.376+0000 7f08ba7fc700 1 -- 172.21.15.57:0/1909192091 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08ac004160 con 0x7f08bc07be60 2023-07-20T18:51:56.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.376+0000 7f08ba7fc700 1 -- 172.21.15.57:0/1909192091 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f08ac0042c0 con 0x7f08bc07be60 2023-07-20T18:51:56.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.376+0000 7f08ba7fc700 1 -- 172.21.15.57:0/1909192091 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08ac011a80 con 0x7f08bc07be60 2023-07-20T18:51:56.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.377+0000 7f08c1c60700 1 -- 172.21.15.57:0/1909192091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 msgr2=0x7f08bc07a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:56.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.377+0000 7f08c1c60700 1 --2- 172.21.15.57:0/1909192091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc07a2c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0098a0 tx=0x7f08ac009bb0 comp rx=0 tx=0).stop 2023-07-20T18:51:56.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.377+0000 7f08c1c60700 1 -- 172.21.15.57:0/1909192091 shutdown_connections 2023-07-20T18:51:56.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.377+0000 7f08c1c60700 1 --2- 172.21.15.57:0/1909192091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc07a2c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:56.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.377+0000 7f08c1c60700 1 -- 172.21.15.57:0/1909192091 >> 172.21.15.57:0/1909192091 conn(0x7f08bc101610 msgr2=0x7f08bc103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:56.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.377+0000 7f08c1c60700 1 -- 172.21.15.57:0/1909192091 shutdown_connections 2023-07-20T18:51:56.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.377+0000 7f08c1c60700 1 -- 172.21.15.57:0/1909192091 wait complete. 2023-07-20T18:51:56.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.379+0000 7f08c1c60700 1 Processor -- start 2023-07-20T18:51:56.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.379+0000 7f08c1c60700 1 -- start start 2023-07-20T18:51:56.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.380+0000 7f08c1c60700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc19be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:56.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.380+0000 7f08c1c60700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc19c3a0 con 0x7f08bc07be60 2023-07-20T18:51:56.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.380+0000 7f08bb7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc19be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:51:56.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.380+0000 7f08bb7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc19be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33030/0 (socket says 172.21.15.57:33030) 2023-07-20T18:51:56.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.380+0000 7f08bb7fe700 1 -- 172.21.15.57:0/1590438538 learned_addr learned my addr 172.21.15.57:0/1590438538 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:51:56.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.380+0000 7f08bb7fe700 1 -- 172.21.15.57:0/1590438538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac009550 con 0x7f08bc07be60 2023-07-20T18:51:56.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.380+0000 7f08bb7fe700 1 --2- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc19be60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f08ac003f20 tx=0x7f08ac011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:51:56.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.381+0000 7f08b8ff9700 1 -- 172.21.15.57:0/1590438538 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08ac003b20 con 0x7f08bc07be60 2023-07-20T18:51:56.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.381+0000 7f08c1c60700 1 -- 172.21.15.57:0/1590438538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08bc19c5a0 con 0x7f08bc07be60 2023-07-20T18:51:56.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.381+0000 7f08b8ff9700 1 -- 172.21.15.57:0/1590438538 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f08ac01a910 con 0x7f08bc07be60 2023-07-20T18:51:56.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.381+0000 7f08b8ff9700 1 -- 172.21.15.57:0/1590438538 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08ac024900 con 0x7f08bc07be60 2023-07-20T18:51:56.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.381+0000 7f08c1c60700 1 -- 172.21.15.57:0/1590438538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc19c9c0 con 0x7f08bc07be60 2023-07-20T18:51:56.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.382+0000 7f08c1c60700 1 -- 172.21.15.57:0/1590438538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc10c4d0 con 0x7f08bc07be60 2023-07-20T18:51:56.121 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.382+0000 7f08b8ff9700 1 -- 172.21.15.57:0/1590438538 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f08ac024b70 con 0x7f08bc07be60 2023-07-20T18:51:56.121 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.383+0000 7f08b8ff9700 1 --2- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f08a4038820 0x7f08a403ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:51:56.121 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.383+0000 7f08b8ff9700 1 -- 172.21.15.57:0/1590438538 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f08ac04ca00 con 0x7f08bc07be60 2023-07-20T18:51:56.121 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.383+0000 7f08baffd700 1 -- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f08a4038820 msgr2=0x7f08a403ace0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2188156324 2023-07-20T18:51:56.121 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.383+0000 7f08baffd700 1 --2- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f08a4038820 0x7f08a403ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T18:51:56.121 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.393+0000 7f08b8ff9700 1 -- 172.21.15.57:0/1590438538 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f08ac018530 con 0x7f08bc07be60 2023-07-20T18:51:56.122 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.584+0000 7f08baffd700 1 -- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f08a4038820 msgr2=0x7f08a403ace0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2188156324 2023-07-20T18:51:56.122 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.584+0000 7f08baffd700 1 --2- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f08a4038820 0x7f08a403ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T18:51:56.122 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.652+0000 7f08c1c60700 1 -- 172.21.15.57:0/1590438538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f08bc10c6c0 con 0x7f08bc07be60 2023-07-20T18:51:56.122 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.652+0000 7f08b8ff9700 1 -- 172.21.15.57:0/1590438538 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f08ac018350 con 0x7f08bc07be60 2023-07-20T18:51:56.122 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.655+0000 7f08c1c60700 1 -- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f08a4038820 msgr2=0x7f08a403ace0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T18:51:56.123 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.655+0000 7f08c1c60700 1 --2- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f08a4038820 0x7f08a403ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:56.123 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.655+0000 7f08c1c60700 1 -- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 msgr2=0x7f08bc19be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:51:56.123 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.655+0000 7f08c1c60700 1 --2- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc19be60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f08ac003f20 tx=0x7f08ac011a50 comp rx=0 tx=0).stop 2023-07-20T18:51:56.123 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.655+0000 7f08c1c60700 1 -- 172.21.15.57:0/1590438538 shutdown_connections 2023-07-20T18:51:56.123 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.655+0000 7f08c1c60700 1 --2- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f08a4038820 0x7f08a403ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:56.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.655+0000 7f08c1c60700 1 --2- 172.21.15.57:0/1590438538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08bc07be60 0x7f08bc19be60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:51:56.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.655+0000 7f08c1c60700 1 -- 172.21.15.57:0/1590438538 >> 172.21.15.57:0/1590438538 conn(0x7f08bc101610 msgr2=0x7f08bc102310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:51:56.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.655+0000 7f08c1c60700 1 -- 172.21.15.57:0/1590438538 shutdown_connections 2023-07-20T18:51:56.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:55.655+0000 7f08c1c60700 1 -- 172.21.15.57:0/1590438538 wait complete. 2023-07-20T18:51:56.124 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for the mgr to restart... 2023-07-20T18:51:56.125 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mgr epoch 9... 2023-07-20T18:51:56.400 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:56 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:56.107+0000 7f042cfd7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T18:51:57.223 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:56 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:56.911+0000 7f042cfd7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T18:51:57.223 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:57 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:57.123+0000 7f042cfd7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T18:51:58.473 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:58.107+0000 7f042cfd7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T18:51:58.473 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:58.231+0000 7f042cfd7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T18:51:58.873 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:58.460+0000 7f042cfd7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T18:51:59.158 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:58.835+0000 7f042cfd7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T18:51:59.473 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:59 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:59.120+0000 7f042cfd7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T18:51:59.473 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:59 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:59.319+0000 7f042cfd7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T18:51:59.973 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:51:59 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:51:59.671+0000 7f042cfd7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T18:52:00.973 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:52:00 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:52:00.631+0000 7f042cfd7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T18:52:00.973 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:52:00 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:52:00.809+0000 7f042cfd7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T18:52:01.473 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:52:01 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:52:01.064+0000 7f042cfd7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T18:52:01.473 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:52:01 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:52:01.216+0000 7f042cfd7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T18:52:01.973 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:52:01 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:52:01.505+0000 7f042cfd7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T18:52:02.973 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:52:02.607+0000 7f042cfd7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T18:52:02.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: Active manager daemon x restarted 2023-07-20T18:52:02.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: Activating manager daemon x 2023-07-20T18:52:02.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: osdmap e3: 0 total, 0 up, 0 in 2023-07-20T18:52:02.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: mgrmap e10: x(active, starting, since 0.00743509s) 2023-07-20T18:52:02.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:52:02.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T18:52:02.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T18:52:02.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T18:52:02.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T18:52:02.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: Manager daemon x is now available 2023-07-20T18:52:02.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:02.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:03.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T18:52:03.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T18:52:03.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:03 smithi057 ceph-mon[126340]: mgrmap e11: x(active, since 1.00912s) 2023-07-20T18:52:04.111 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2023-07-20T18:52:04.111 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-20T18:52:04.111 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T18:52:04.112 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2023-07-20T18:52:04.115 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.633+0000 7f122512c700 1 Processor -- start 2023-07-20T18:52:04.115 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.633+0000 7f122512c700 1 -- start start 2023-07-20T18:52:04.115 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.633+0000 7f122512c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122007a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:04.115 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.633+0000 7f122512c700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122007a8b0 con 0x7f122007be80 2023-07-20T18:52:04.115 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.634+0000 7f121ed9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122007a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:04.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.634+0000 7f121ed9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122007a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33046/0 (socket says 172.21.15.57:33046) 2023-07-20T18:52:04.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.634+0000 7f121ed9d700 1 -- 172.21.15.57:0/874032738 learned_addr learned my addr 172.21.15.57:0/874032738 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:04.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.634+0000 7f121ed9d700 1 -- 172.21.15.57:0/874032738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122007a9f0 con 0x7f122007be80 2023-07-20T18:52:04.116 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.634+0000 7f121ed9d700 1 --2- 172.21.15.57:0/874032738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122007a2e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12100098a0 tx=0x7f1210009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46f71fdaf8eb3b2 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:04.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.635+0000 7f121dd9b700 1 -- 172.21.15.57:0/874032738 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1210004160 con 0x7f122007be80 2023-07-20T18:52:04.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.635+0000 7f121dd9b700 1 -- 172.21.15.57:0/874032738 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f12100042c0 con 0x7f122007be80 2023-07-20T18:52:04.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.635+0000 7f121dd9b700 1 -- 172.21.15.57:0/874032738 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1210011a80 con 0x7f122007be80 2023-07-20T18:52:04.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.636+0000 7f122512c700 1 -- 172.21.15.57:0/874032738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 msgr2=0x7f122007a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:04.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.636+0000 7f122512c700 1 --2- 172.21.15.57:0/874032738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122007a2e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12100098a0 tx=0x7f1210009bb0 comp rx=0 tx=0).stop 2023-07-20T18:52:04.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.636+0000 7f122512c700 1 -- 172.21.15.57:0/874032738 shutdown_connections 2023-07-20T18:52:04.117 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.636+0000 7f122512c700 1 --2- 172.21.15.57:0/874032738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122007a2e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:04.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.636+0000 7f122512c700 1 -- 172.21.15.57:0/874032738 >> 172.21.15.57:0/874032738 conn(0x7f1220101610 msgr2=0x7f1220103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:04.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.636+0000 7f122512c700 1 -- 172.21.15.57:0/874032738 shutdown_connections 2023-07-20T18:52:04.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.636+0000 7f122512c700 1 -- 172.21.15.57:0/874032738 wait complete. 2023-07-20T18:52:04.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.637+0000 7f122512c700 1 Processor -- start 2023-07-20T18:52:04.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.637+0000 7f122512c700 1 -- start start 2023-07-20T18:52:04.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.637+0000 7f122512c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122019be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:04.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.637+0000 7f122512c700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122019c3b0 con 0x7f122007be80 2023-07-20T18:52:04.118 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.638+0000 7f121ed9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122019be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:04.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.638+0000 7f121ed9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122019be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33052/0 (socket says 172.21.15.57:33052) 2023-07-20T18:52:04.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.638+0000 7f121ed9d700 1 -- 172.21.15.57:0/1825128618 learned_addr learned my addr 172.21.15.57:0/1825128618 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:04.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.638+0000 7f121ed9d700 1 -- 172.21.15.57:0/1825128618 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1210009550 con 0x7f122007be80 2023-07-20T18:52:04.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.638+0000 7f121ed9d700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122019be70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1210003f20 tx=0x7f1210011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:04.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.639+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1210003b20 con 0x7f122007be80 2023-07-20T18:52:04.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.639+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f121001a910 con 0x7f122007be80 2023-07-20T18:52:04.119 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.639+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12100248c0 con 0x7f122007be80 2023-07-20T18:52:04.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.639+0000 7f122512c700 1 -- 172.21.15.57:0/1825128618 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122019c5b0 con 0x7f122007be80 2023-07-20T18:52:04.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.639+0000 7f122512c700 1 -- 172.21.15.57:0/1825128618 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122019c9d0 con 0x7f122007be80 2023-07-20T18:52:04.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.640+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f121001aa80 con 0x7f122007be80 2023-07-20T18:52:04.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.640+0000 7f1207fff700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 0x7f120803ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:04.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.640+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 --> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f120803b440 con 0x7f1208038870 2023-07-20T18:52:04.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.640+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f121004cab0 con 0x7f122007be80 2023-07-20T18:52:04.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.641+0000 7f121e59c700 1 -- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 msgr2=0x7f120803ad30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2188156324 2023-07-20T18:52:04.120 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.641+0000 7f121e59c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 0x7f120803ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T18:52:04.121 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.842+0000 7f121e59c700 1 -- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 msgr2=0x7f120803ad30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2188156324 2023-07-20T18:52:04.121 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:56.842+0000 7f121e59c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 0x7f120803ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T18:52:04.123 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:57.243+0000 7f121e59c700 1 -- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 msgr2=0x7f120803ad30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2188156324 2023-07-20T18:52:04.123 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:57.243+0000 7f121e59c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 0x7f120803ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T18:52:04.123 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:58.044+0000 7f121e59c700 1 -- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 msgr2=0x7f120803ad30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2188156324 2023-07-20T18:52:04.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:58.044+0000 7f121e59c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 0x7f120803ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T18:52:04.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:59.644+0000 7f121e59c700 1 -- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 msgr2=0x7f120803ad30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2188156324 2023-07-20T18:52:04.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:51:59.644+0000 7f121e59c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 0x7f120803ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T18:52:04.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:02.615+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f121001a430 con 0x7f122007be80 2023-07-20T18:52:04.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:02.615+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 msgr2=0x7f120803ad30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T18:52:04.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:02.615+0000 7f1207fff700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 0x7f120803ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:04.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.616+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f121004e5f0 con 0x7f122007be80 2023-07-20T18:52:04.124 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.616+0000 7f1207fff700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f120803c1b0 0x7f120803e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:04.125 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.616+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f120803b440 con 0x7f120803c1b0 2023-07-20T18:52:04.125 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.617+0000 7f121e59c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f120803c1b0 0x7f120803e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:04.125 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.618+0000 7f121e59c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f120803c1b0 0x7f120803e5a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1214003a10 tx=0x7f1214009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:04.125 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.619+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f120803b440 con 0x7f120803c1b0 2023-07-20T18:52:04.125 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.626+0000 7f122512c700 1 -- 172.21.15.57:0/1825128618 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f122019d1b0 con 0x7f120803c1b0 2023-07-20T18:52:04.125 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.626+0000 7f1207fff700 1 -- 172.21.15.57:0/1825128618 <== mgr.14150 v2:172.21.15.57:6800/4059331298 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f122019d1b0 con 0x7f120803c1b0 2023-07-20T18:52:04.125 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 -- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f120803c1b0 msgr2=0x7f120803e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:04.126 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f120803c1b0 0x7f120803e5a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1214003a10 tx=0x7f1214009270 comp rx=0 tx=0).stop 2023-07-20T18:52:04.126 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 -- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 msgr2=0x7f122019be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:04.126 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122019be70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1210003f20 tx=0x7f1210011b00 comp rx=0 tx=0).stop 2023-07-20T18:52:04.126 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 -- 172.21.15.57:0/1825128618 shutdown_connections 2023-07-20T18:52:04.126 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f120803c1b0 0x7f120803e5a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:04.126 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f122007be80 0x7f122019be70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:04.126 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 --2- 172.21.15.57:0/1825128618 >> [v2:172.21.15.57:6800/2188156324,v1:172.21.15.57:6801/2188156324] conn(0x7f1208038870 0x7f120803ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:04.126 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 -- 172.21.15.57:0/1825128618 >> 172.21.15.57:0/1825128618 conn(0x7f1220101610 msgr2=0x7f12201022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:04.127 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 -- 172.21.15.57:0/1825128618 shutdown_connections 2023-07-20T18:52:04.127 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:03.627+0000 7f122512c700 1 -- 172.21.15.57:0/1825128618 wait complete. 2023-07-20T18:52:04.127 INFO:teuthology.orchestra.run.smithi057.stdout:mgr epoch 9 is available 2023-07-20T18:52:04.127 INFO:teuthology.orchestra.run.smithi057.stdout:Generating a dashboard self-signed certificate... 2023-07-20T18:52:04.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:04 smithi057 ceph-mon[126340]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T18:52:04.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:04 smithi057 ceph-mon[126340]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T18:52:05.715 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-20T18:52:05.716 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.889+0000 7f0d90f3e700 1 Processor -- start 2023-07-20T18:52:05.716 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.889+0000 7f0d90f3e700 1 -- start start 2023-07-20T18:52:05.716 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.890+0000 7f0d90f3e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:05.717 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.890+0000 7f0d90f3e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c106a60 con 0x7f0d8c106070 2023-07-20T18:52:05.717 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.890+0000 7f0d8a59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:05.717 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.890+0000 7f0d8a59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40924/0 (socket says 172.21.15.57:40924) 2023-07-20T18:52:05.717 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.890+0000 7f0d8a59c700 1 -- 172.21.15.57:0/3664690892 learned_addr learned my addr 172.21.15.57:0/3664690892 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:05.717 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.890+0000 7f0d8a59c700 1 -- 172.21.15.57:0/3664690892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c107270 con 0x7f0d8c106070 2023-07-20T18:52:05.717 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.891+0000 7f0d8a59c700 1 --2- 172.21.15.57:0/3664690892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c106490 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0098a0 tx=0x7f0d7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dc8083a781cf2e server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:05.717 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.891+0000 7f0d8959a700 1 -- 172.21.15.57:0/3664690892 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d7c004160 con 0x7f0d8c106070 2023-07-20T18:52:05.718 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.891+0000 7f0d8959a700 1 -- 172.21.15.57:0/3664690892 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f0d7c0042c0 con 0x7f0d8c106070 2023-07-20T18:52:05.718 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.891+0000 7f0d8959a700 1 -- 172.21.15.57:0/3664690892 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d7c011a80 con 0x7f0d8c106070 2023-07-20T18:52:05.718 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.892+0000 7f0d90f3e700 1 -- 172.21.15.57:0/3664690892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 msgr2=0x7f0d8c106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:05.718 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.892+0000 7f0d90f3e700 1 --2- 172.21.15.57:0/3664690892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c106490 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0098a0 tx=0x7f0d7c009bb0 comp rx=0 tx=0).stop 2023-07-20T18:52:05.718 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.892+0000 7f0d90f3e700 1 -- 172.21.15.57:0/3664690892 shutdown_connections 2023-07-20T18:52:05.718 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.892+0000 7f0d90f3e700 1 --2- 172.21.15.57:0/3664690892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c106490 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:05.718 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.892+0000 7f0d90f3e700 1 -- 172.21.15.57:0/3664690892 >> 172.21.15.57:0/3664690892 conn(0x7f0d8c101610 msgr2=0x7f0d8c103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:05.718 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.892+0000 7f0d90f3e700 1 -- 172.21.15.57:0/3664690892 shutdown_connections 2023-07-20T18:52:05.719 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.892+0000 7f0d90f3e700 1 -- 172.21.15.57:0/3664690892 wait complete. 2023-07-20T18:52:05.719 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.893+0000 7f0d90f3e700 1 Processor -- start 2023-07-20T18:52:05.719 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.893+0000 7f0d90f3e700 1 -- start start 2023-07-20T18:52:05.719 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.893+0000 7f0d90f3e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:05.719 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.893+0000 7f0d90f3e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c19c430 con 0x7f0d8c106070 2023-07-20T18:52:05.719 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.894+0000 7f0d8a59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:05.719 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.894+0000 7f0d8a59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c19bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40938/0 (socket says 172.21.15.57:40938) 2023-07-20T18:52:05.720 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.894+0000 7f0d8a59c700 1 -- 172.21.15.57:0/2469651598 learned_addr learned my addr 172.21.15.57:0/2469651598 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:05.720 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.894+0000 7f0d8a59c700 1 -- 172.21.15.57:0/2469651598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c009550 con 0x7f0d8c106070 2023-07-20T18:52:05.720 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.894+0000 7f0d8a59c700 1 --2- 172.21.15.57:0/2469651598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c19bef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c003f20 tx=0x7f0d7c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:05.720 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.894+0000 7f0d7b7fe700 1 -- 172.21.15.57:0/2469651598 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d7c0039e0 con 0x7f0d8c106070 2023-07-20T18:52:05.720 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.894+0000 7f0d7b7fe700 1 -- 172.21.15.57:0/2469651598 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f0d7c003b40 con 0x7f0d8c106070 2023-07-20T18:52:05.720 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.894+0000 7f0d7b7fe700 1 -- 172.21.15.57:0/2469651598 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d7c024710 con 0x7f0d8c106070 2023-07-20T18:52:05.720 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.894+0000 7f0d90f3e700 1 -- 172.21.15.57:0/2469651598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c19c630 con 0x7f0d8c106070 2023-07-20T18:52:05.721 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.895+0000 7f0d90f3e700 1 -- 172.21.15.57:0/2469651598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c19ca50 con 0x7f0d8c106070 2023-07-20T18:52:05.721 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.896+0000 7f0d90f3e700 1 -- 172.21.15.57:0/2469651598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8c061da0 con 0x7f0d8c106070 2023-07-20T18:52:05.721 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.896+0000 7f0d7b7fe700 1 -- 172.21.15.57:0/2469651598 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f0d7c01ab30 con 0x7f0d8c106070 2023-07-20T18:52:05.721 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.896+0000 7f0d7b7fe700 1 --2- 172.21.15.57:0/2469651598 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0d74038820 0x7f0d7403ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:05.721 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.896+0000 7f0d7b7fe700 1 -- 172.21.15.57:0/2469651598 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0d7c04c940 con 0x7f0d8c106070 2023-07-20T18:52:05.721 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.896+0000 7f0d89d9b700 1 --2- 172.21.15.57:0/2469651598 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0d74038820 0x7f0d7403ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:05.721 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.897+0000 7f0d89d9b700 1 --2- 172.21.15.57:0/2469651598 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0d74038820 0x7f0d7403ace0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0d80006fd0 tx=0x7f0d80006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:05.721 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:04.901+0000 7f0d7b7fe700 1 -- 172.21.15.57:0/2469651598 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0d7c01a950 con 0x7f0d8c106070 2023-07-20T18:52:05.722 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.124+0000 7f0d90f3e700 1 -- 172.21.15.57:0/2469651598 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d8c102a90 con 0x7f0d74038820 2023-07-20T18:52:05.722 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.215+0000 7f0d7b7fe700 1 -- 172.21.15.57:0/2469651598 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f0d8c102a90 con 0x7f0d74038820 2023-07-20T18:52:05.722 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.218+0000 7f0d90f3e700 1 -- 172.21.15.57:0/2469651598 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0d74038820 msgr2=0x7f0d7403ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:05.722 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.218+0000 7f0d90f3e700 1 --2- 172.21.15.57:0/2469651598 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0d74038820 0x7f0d7403ace0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0d80006fd0 tx=0x7f0d80006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:05.722 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.218+0000 7f0d90f3e700 1 -- 172.21.15.57:0/2469651598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 msgr2=0x7f0d8c19bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:05.722 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.218+0000 7f0d90f3e700 1 --2- 172.21.15.57:0/2469651598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c19bef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c003f20 tx=0x7f0d7c011b00 comp rx=0 tx=0).stop 2023-07-20T18:52:05.722 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.218+0000 7f0d90f3e700 1 -- 172.21.15.57:0/2469651598 shutdown_connections 2023-07-20T18:52:05.723 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.218+0000 7f0d90f3e700 1 --2- 172.21.15.57:0/2469651598 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0d74038820 0x7f0d7403ace0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:05.723 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.218+0000 7f0d90f3e700 1 --2- 172.21.15.57:0/2469651598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d8c106070 0x7f0d8c19bef0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:05.723 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.218+0000 7f0d90f3e700 1 -- 172.21.15.57:0/2469651598 >> 172.21.15.57:0/2469651598 conn(0x7f0d8c101610 msgr2=0x7f0d8c102380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:05.723 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.218+0000 7f0d90f3e700 1 -- 172.21.15.57:0/2469651598 shutdown_connections 2023-07-20T18:52:05.723 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:05.218+0000 7f0d90f3e700 1 -- 172.21.15.57:0/2469651598 wait complete. 2023-07-20T18:52:05.723 INFO:teuthology.orchestra.run.smithi057.stdout:Creating initial admin user... 2023-07-20T18:52:05.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:05 smithi057 ceph-mon[126340]: [20/Jul/2023:18:52:04] ENGINE Bus STARTING 2023-07-20T18:52:05.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:05 smithi057 ceph-mon[126340]: [20/Jul/2023:18:52:04] ENGINE Serving on http://172.21.15.57:8765 2023-07-20T18:52:05.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:05 smithi057 ceph-mon[126340]: [20/Jul/2023:18:52:04] ENGINE Serving on https://172.21.15.57:7150 2023-07-20T18:52:05.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:05 smithi057 ceph-mon[126340]: [20/Jul/2023:18:52:04] ENGINE Bus STARTED 2023-07-20T18:52:05.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:05 smithi057 ceph-mon[126340]: mgrmap e12: x(active, since 2s) 2023-07-20T18:52:05.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:05 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:05.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:05 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:05.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:05 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:06.963 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:06 smithi057 ceph-mon[126340]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:52:06.963 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:07.195 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$xtst7pjwYjKvpVEu9LiJO.lfwyhDi5c6b27OUZMZTQHQMTueKXr2q", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689879126, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-20T18:52:07.195 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.272+0000 7ff02cd8f700 1 Processor -- start 2023-07-20T18:52:07.195 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.272+0000 7ff02cd8f700 1 -- start start 2023-07-20T18:52:07.196 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.272+0000 7ff02cd8f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02807a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:07.196 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.272+0000 7ff02cd8f700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02807a8b0 con 0x7ff02807be80 2023-07-20T18:52:07.196 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.273+0000 7ff02659c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02807a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:07.196 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.273+0000 7ff02659c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02807a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40944/0 (socket says 172.21.15.57:40944) 2023-07-20T18:52:07.196 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.273+0000 7ff02659c700 1 -- 172.21.15.57:0/3288685092 learned_addr learned my addr 172.21.15.57:0/3288685092 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:07.196 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.273+0000 7ff02659c700 1 -- 172.21.15.57:0/3288685092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02807a9f0 con 0x7ff02807be80 2023-07-20T18:52:07.196 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.273+0000 7ff02659c700 1 --2- 172.21.15.57:0/3288685092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02807a2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff0180098a0 tx=0x7ff018009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a12f7b51e704bc server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:07.197 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.274+0000 7ff02559a700 1 -- 172.21.15.57:0/3288685092 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff018004160 con 0x7ff02807be80 2023-07-20T18:52:07.197 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.274+0000 7ff02559a700 1 -- 172.21.15.57:0/3288685092 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7ff0180042c0 con 0x7ff02807be80 2023-07-20T18:52:07.197 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.274+0000 7ff02559a700 1 -- 172.21.15.57:0/3288685092 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff018011a80 con 0x7ff02807be80 2023-07-20T18:52:07.197 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.274+0000 7ff02cd8f700 1 -- 172.21.15.57:0/3288685092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 msgr2=0x7ff02807a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:07.197 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.274+0000 7ff02cd8f700 1 --2- 172.21.15.57:0/3288685092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02807a2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff0180098a0 tx=0x7ff018009bb0 comp rx=0 tx=0).stop 2023-07-20T18:52:07.197 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.274+0000 7ff02cd8f700 1 -- 172.21.15.57:0/3288685092 shutdown_connections 2023-07-20T18:52:07.197 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.274+0000 7ff02cd8f700 1 --2- 172.21.15.57:0/3288685092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02807a2e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:07.197 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.274+0000 7ff02cd8f700 1 -- 172.21.15.57:0/3288685092 >> 172.21.15.57:0/3288685092 conn(0x7ff028101610 msgr2=0x7ff028103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:07.198 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.274+0000 7ff02cd8f700 1 -- 172.21.15.57:0/3288685092 shutdown_connections 2023-07-20T18:52:07.198 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.275+0000 7ff02cd8f700 1 -- 172.21.15.57:0/3288685092 wait complete. 2023-07-20T18:52:07.198 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.276+0000 7ff02cd8f700 1 Processor -- start 2023-07-20T18:52:07.198 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.276+0000 7ff02cd8f700 1 -- start start 2023-07-20T18:52:07.198 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.277+0000 7ff02cd8f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02819be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:07.198 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.277+0000 7ff02cd8f700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02819c3b0 con 0x7ff02807be80 2023-07-20T18:52:07.198 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.277+0000 7ff02659c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02819be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:07.199 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.277+0000 7ff02659c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02819be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40958/0 (socket says 172.21.15.57:40958) 2023-07-20T18:52:07.199 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.277+0000 7ff02659c700 1 -- 172.21.15.57:0/439337915 learned_addr learned my addr 172.21.15.57:0/439337915 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:07.199 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.277+0000 7ff02659c700 1 -- 172.21.15.57:0/439337915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff018009550 con 0x7ff02807be80 2023-07-20T18:52:07.199 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.277+0000 7ff02659c700 1 --2- 172.21.15.57:0/439337915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02819be70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff018003f20 tx=0x7ff018011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:07.199 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.278+0000 7ff0177fe700 1 -- 172.21.15.57:0/439337915 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff018003b20 con 0x7ff02807be80 2023-07-20T18:52:07.199 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.278+0000 7ff0177fe700 1 -- 172.21.15.57:0/439337915 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7ff01801a910 con 0x7ff02807be80 2023-07-20T18:52:07.199 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.278+0000 7ff02cd8f700 1 -- 172.21.15.57:0/439337915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff02819c5b0 con 0x7ff02807be80 2023-07-20T18:52:07.199 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.278+0000 7ff0177fe700 1 -- 172.21.15.57:0/439337915 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff0180248c0 con 0x7ff02807be80 2023-07-20T18:52:07.200 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.278+0000 7ff02cd8f700 1 -- 172.21.15.57:0/439337915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff02819c9d0 con 0x7ff02807be80 2023-07-20T18:52:07.200 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.279+0000 7ff0177fe700 1 -- 172.21.15.57:0/439337915 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7ff01801aa80 con 0x7ff02807be80 2023-07-20T18:52:07.200 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.279+0000 7ff0177fe700 1 --2- 172.21.15.57:0/439337915 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff010038820 0x7ff01003ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:07.200 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.279+0000 7ff0177fe700 1 -- 172.21.15.57:0/439337915 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff01804cc60 con 0x7ff02807be80 2023-07-20T18:52:07.200 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.279+0000 7ff02cd8f700 1 -- 172.21.15.57:0/439337915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff028061da0 con 0x7ff02807be80 2023-07-20T18:52:07.200 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.280+0000 7ff025d9b700 1 --2- 172.21.15.57:0/439337915 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff010038820 0x7ff01003ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:07.200 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.280+0000 7ff025d9b700 1 --2- 172.21.15.57:0/439337915 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff010038820 0x7ff01003ace0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff01c006fd0 tx=0x7ff01c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:07.201 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.285+0000 7ff0177fe700 1 -- 172.21.15.57:0/439337915 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff01801adf0 con 0x7ff02807be80 2023-07-20T18:52:07.201 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.517+0000 7ff02cd8f700 1 -- 172.21.15.57:0/439337915 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff02819f4d0 con 0x7ff010038820 2023-07-20T18:52:07.201 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.768+0000 7ff0177fe700 1 -- 172.21.15.57:0/439337915 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7ff02819f4d0 con 0x7ff010038820 2023-07-20T18:52:07.201 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.770+0000 7ff02cd8f700 1 -- 172.21.15.57:0/439337915 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff010038820 msgr2=0x7ff01003ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:07.201 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.770+0000 7ff02cd8f700 1 --2- 172.21.15.57:0/439337915 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff010038820 0x7ff01003ace0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff01c006fd0 tx=0x7ff01c006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:07.201 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.770+0000 7ff02cd8f700 1 -- 172.21.15.57:0/439337915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 msgr2=0x7ff02819be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:07.201 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.770+0000 7ff02cd8f700 1 --2- 172.21.15.57:0/439337915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02819be70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff018003f20 tx=0x7ff018011b00 comp rx=0 tx=0).stop 2023-07-20T18:52:07.201 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.770+0000 7ff02cd8f700 1 -- 172.21.15.57:0/439337915 shutdown_connections 2023-07-20T18:52:07.202 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.770+0000 7ff02cd8f700 1 --2- 172.21.15.57:0/439337915 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff010038820 0x7ff01003ace0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:07.202 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.770+0000 7ff02cd8f700 1 --2- 172.21.15.57:0/439337915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02807be80 0x7ff02819be70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:07.202 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.770+0000 7ff02cd8f700 1 -- 172.21.15.57:0/439337915 >> 172.21.15.57:0/439337915 conn(0x7ff028101610 msgr2=0x7ff0281022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:07.202 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.770+0000 7ff02cd8f700 1 -- 172.21.15.57:0/439337915 shutdown_connections 2023-07-20T18:52:07.202 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:06.770+0000 7ff02cd8f700 1 -- 172.21.15.57:0/439337915 wait complete. 2023-07-20T18:52:07.202 INFO:teuthology.orchestra.run.smithi057.stdout:Fetching dashboard port number... 2023-07-20T18:52:08.070 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:07 smithi057 ceph-mon[126340]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:52:08.071 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:08.582 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 8443 2023-07-20T18:52:08.582 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.718+0000 7f65b3f9e700 1 Processor -- start 2023-07-20T18:52:08.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.718+0000 7f65b3f9e700 1 -- start start 2023-07-20T18:52:08.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.719+0000 7f65b3f9e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac07a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:08.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.719+0000 7f65b3f9e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac07a8f0 con 0x7f65ac07bec0 2023-07-20T18:52:08.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.719+0000 7f65b1d3a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac07a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:08.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.719+0000 7f65b1d3a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac07a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40964/0 (socket says 172.21.15.57:40964) 2023-07-20T18:52:08.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.719+0000 7f65b1d3a700 1 -- 172.21.15.57:0/2099457466 learned_addr learned my addr 172.21.15.57:0/2099457466 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:08.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.720+0000 7f65b1d3a700 1 -- 172.21.15.57:0/2099457466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac07aa30 con 0x7f65ac07bec0 2023-07-20T18:52:08.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.720+0000 7f65b1d3a700 1 --2- 172.21.15.57:0/2099457466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac07a320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f65a80098a0 tx=0x7f65a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e090cb05057db7 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:08.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.720+0000 7f65b0d38700 1 -- 172.21.15.57:0/2099457466 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65a8004160 con 0x7f65ac07bec0 2023-07-20T18:52:08.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.721+0000 7f65b0d38700 1 -- 172.21.15.57:0/2099457466 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f65a80042c0 con 0x7f65ac07bec0 2023-07-20T18:52:08.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.721+0000 7f65b0d38700 1 -- 172.21.15.57:0/2099457466 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65a8011a80 con 0x7f65ac07bec0 2023-07-20T18:52:08.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.721+0000 7f65b3f9e700 1 -- 172.21.15.57:0/2099457466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 msgr2=0x7f65ac07a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:08.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.721+0000 7f65b3f9e700 1 --2- 172.21.15.57:0/2099457466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac07a320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f65a80098a0 tx=0x7f65a8009bb0 comp rx=0 tx=0).stop 2023-07-20T18:52:08.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.722+0000 7f65b3f9e700 1 -- 172.21.15.57:0/2099457466 shutdown_connections 2023-07-20T18:52:08.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.722+0000 7f65b3f9e700 1 --2- 172.21.15.57:0/2099457466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac07a320 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:08.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.722+0000 7f65b3f9e700 1 -- 172.21.15.57:0/2099457466 >> 172.21.15.57:0/2099457466 conn(0x7f65ac101610 msgr2=0x7f65ac103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:08.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.722+0000 7f65b3f9e700 1 -- 172.21.15.57:0/2099457466 shutdown_connections 2023-07-20T18:52:08.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.722+0000 7f65b3f9e700 1 -- 172.21.15.57:0/2099457466 wait complete. 2023-07-20T18:52:08.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.723+0000 7f65b3f9e700 1 Processor -- start 2023-07-20T18:52:08.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.723+0000 7f65b3f9e700 1 -- start start 2023-07-20T18:52:08.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.724+0000 7f65b3f9e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac1093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:08.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.724+0000 7f65b3f9e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac109900 con 0x7f65ac07bec0 2023-07-20T18:52:08.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.725+0000 7f65b1d3a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac1093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:08.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.725+0000 7f65b1d3a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac1093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40978/0 (socket says 172.21.15.57:40978) 2023-07-20T18:52:08.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.725+0000 7f65b1d3a700 1 -- 172.21.15.57:0/1594422351 learned_addr learned my addr 172.21.15.57:0/1594422351 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:08.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.725+0000 7f65b1d3a700 1 -- 172.21.15.57:0/1594422351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a8009550 con 0x7f65ac07bec0 2023-07-20T18:52:08.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.726+0000 7f65b1d3a700 1 --2- 172.21.15.57:0/1594422351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac1093c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65a8003f20 tx=0x7f65a8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:08.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.726+0000 7f65a2ffd700 1 -- 172.21.15.57:0/1594422351 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65a8003b20 con 0x7f65ac07bec0 2023-07-20T18:52:08.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.726+0000 7f65a2ffd700 1 -- 172.21.15.57:0/1594422351 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f65a801a910 con 0x7f65ac07bec0 2023-07-20T18:52:08.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.726+0000 7f65a2ffd700 1 -- 172.21.15.57:0/1594422351 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65a8024850 con 0x7f65ac07bec0 2023-07-20T18:52:08.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.726+0000 7f65b3f9e700 1 -- 172.21.15.57:0/1594422351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ac109b00 con 0x7f65ac07bec0 2023-07-20T18:52:08.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.726+0000 7f65b3f9e700 1 -- 172.21.15.57:0/1594422351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ac105ff0 con 0x7f65ac07bec0 2023-07-20T18:52:08.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.727+0000 7f65a2ffd700 1 -- 172.21.15.57:0/1594422351 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f65a802d3f0 con 0x7f65ac07bec0 2023-07-20T18:52:08.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.728+0000 7f65b3f9e700 1 -- 172.21.15.57:0/1594422351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ac061da0 con 0x7f65ac07bec0 2023-07-20T18:52:08.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.728+0000 7f65a2ffd700 1 --2- 172.21.15.57:0/1594422351 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6598038870 0x7f659803ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:08.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.728+0000 7f65a2ffd700 1 -- 172.21.15.57:0/1594422351 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f65a804cb90 con 0x7f65ac07bec0 2023-07-20T18:52:08.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.728+0000 7f65b1539700 1 --2- 172.21.15.57:0/1594422351 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6598038870 0x7f659803ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:08.588 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.729+0000 7f65b1539700 1 --2- 172.21.15.57:0/1594422351 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6598038870 0x7f659803ad30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f659c006fd0 tx=0x7f659c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:08.588 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.734+0000 7f65a2ffd700 1 -- 172.21.15.57:0/1594422351 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f65a80109d0 con 0x7f65ac07bec0 2023-07-20T18:52:08.588 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.955+0000 7f65b3f9e700 1 -- 172.21.15.57:0/1594422351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f65ac106730 con 0x7f65ac07bec0 2023-07-20T18:52:08.588 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.955+0000 7f65a2ffd700 1 -- 172.21.15.57:0/1594422351 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f65a80107f0 con 0x7f65ac07bec0 2023-07-20T18:52:08.588 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.957+0000 7f65b3f9e700 1 -- 172.21.15.57:0/1594422351 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6598038870 msgr2=0x7f659803ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:08.588 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.957+0000 7f65b3f9e700 1 --2- 172.21.15.57:0/1594422351 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6598038870 0x7f659803ad30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f659c006fd0 tx=0x7f659c006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:08.588 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.957+0000 7f65b3f9e700 1 -- 172.21.15.57:0/1594422351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 msgr2=0x7f65ac1093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:08.589 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.957+0000 7f65b3f9e700 1 --2- 172.21.15.57:0/1594422351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac1093c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65a8003f20 tx=0x7f65a8011b00 comp rx=0 tx=0).stop 2023-07-20T18:52:08.589 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.957+0000 7f65b3f9e700 1 -- 172.21.15.57:0/1594422351 shutdown_connections 2023-07-20T18:52:08.589 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.957+0000 7f65b3f9e700 1 --2- 172.21.15.57:0/1594422351 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6598038870 0x7f659803ad30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:08.589 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.957+0000 7f65b3f9e700 1 --2- 172.21.15.57:0/1594422351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65ac07bec0 0x7f65ac1093c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:08.589 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.957+0000 7f65b3f9e700 1 -- 172.21.15.57:0/1594422351 >> 172.21.15.57:0/1594422351 conn(0x7f65ac101610 msgr2=0x7f65ac102380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:08.589 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.958+0000 7f65b3f9e700 1 -- 172.21.15.57:0/1594422351 shutdown_connections 2023-07-20T18:52:08.589 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:07.958+0000 7f65b3f9e700 1 -- 172.21.15.57:0/1594422351 wait complete. 2023-07-20T18:52:08.601 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T18:52:08.601 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout disabled 2023-07-20T18:52:08.618 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T18:52:08.618 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2023-07-20T18:52:08.618 INFO:teuthology.orchestra.run.smithi057.stdout:firewalld.service is not enabled 2023-07-20T18:52:08.618 INFO:teuthology.orchestra.run.smithi057.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-20T18:52:08.620 INFO:teuthology.orchestra.run.smithi057.stdout:Ceph Dashboard is now available at: 2023-07-20T18:52:08.620 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:08.620 INFO:teuthology.orchestra.run.smithi057.stdout: URL: https://smithi057.front.sepia.ceph.com:8443/ 2023-07-20T18:52:08.620 INFO:teuthology.orchestra.run.smithi057.stdout: User: admin 2023-07-20T18:52:08.621 INFO:teuthology.orchestra.run.smithi057.stdout: Password: s5v50tfxs3 2023-07-20T18:52:08.621 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:08.621 INFO:teuthology.orchestra.run.smithi057.stdout:Saving cluster configuration to /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config directory 2023-07-20T18:52:08.621 INFO:teuthology.orchestra.run.smithi057.stdout:Enabling autotune for osd_memory_target 2023-07-20T18:52:09.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:08 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1594422351' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-20T18:52:09.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:09 smithi057 ceph-mon[126340]: mgrmap e13: x(active, since 6s) 2023-07-20T18:52:10.075 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.368+0000 7fc3b0cc1700 1 Processor -- start 2023-07-20T18:52:10.075 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.368+0000 7fc3b0cc1700 1 -- start start 2023-07-20T18:52:10.075 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.368+0000 7fc3b0cc1700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:10.075 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.368+0000 7fc3b0cc1700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac106a60 con 0x7fc3ac106070 2023-07-20T18:52:10.076 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.369+0000 7fc3aa59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:10.076 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.369+0000 7fc3aa59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40988/0 (socket says 172.21.15.57:40988) 2023-07-20T18:52:10.076 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.369+0000 7fc3aa59c700 1 -- 172.21.15.57:0/2646127935 learned_addr learned my addr 172.21.15.57:0/2646127935 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:10.076 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.369+0000 7fc3aa59c700 1 -- 172.21.15.57:0/2646127935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac107270 con 0x7fc3ac106070 2023-07-20T18:52:10.076 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.370+0000 7fc3aa59c700 1 --2- 172.21.15.57:0/2646127935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac106490 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc39c0098a0 tx=0x7fc39c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ce16b3ad3322ad server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:10.076 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.370+0000 7fc3a959a700 1 -- 172.21.15.57:0/2646127935 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc39c004160 con 0x7fc3ac106070 2023-07-20T18:52:10.077 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.370+0000 7fc3a959a700 1 -- 172.21.15.57:0/2646127935 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fc39c0042c0 con 0x7fc3ac106070 2023-07-20T18:52:10.077 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.370+0000 7fc3a959a700 1 -- 172.21.15.57:0/2646127935 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc39c011a80 con 0x7fc3ac106070 2023-07-20T18:52:10.077 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.370+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/2646127935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 msgr2=0x7fc3ac106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:10.077 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.370+0000 7fc3b0cc1700 1 --2- 172.21.15.57:0/2646127935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac106490 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc39c0098a0 tx=0x7fc39c009bb0 comp rx=0 tx=0).stop 2023-07-20T18:52:10.077 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.371+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/2646127935 shutdown_connections 2023-07-20T18:52:10.077 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.371+0000 7fc3b0cc1700 1 --2- 172.21.15.57:0/2646127935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac106490 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:10.077 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.371+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/2646127935 >> 172.21.15.57:0/2646127935 conn(0x7fc3ac101610 msgr2=0x7fc3ac103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:10.077 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.371+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/2646127935 shutdown_connections 2023-07-20T18:52:10.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.371+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/2646127935 wait complete. 2023-07-20T18:52:10.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.372+0000 7fc3b0cc1700 1 Processor -- start 2023-07-20T18:52:10.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.372+0000 7fc3b0cc1700 1 -- start start 2023-07-20T18:52:10.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.372+0000 7fc3b0cc1700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:10.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.372+0000 7fc3b0cc1700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac19c430 con 0x7fc3ac106070 2023-07-20T18:52:10.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.372+0000 7fc3aa59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:10.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.372+0000 7fc3aa59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac19bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40996/0 (socket says 172.21.15.57:40996) 2023-07-20T18:52:10.078 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.372+0000 7fc3aa59c700 1 -- 172.21.15.57:0/1300764540 learned_addr learned my addr 172.21.15.57:0/1300764540 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:10.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.373+0000 7fc3aa59c700 1 -- 172.21.15.57:0/1300764540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c009550 con 0x7fc3ac106070 2023-07-20T18:52:10.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.373+0000 7fc3aa59c700 1 --2- 172.21.15.57:0/1300764540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac19bef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc39c003f20 tx=0x7fc39c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:10.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.373+0000 7fc39b7fe700 1 -- 172.21.15.57:0/1300764540 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc39c0039e0 con 0x7fc3ac106070 2023-07-20T18:52:10.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.373+0000 7fc39b7fe700 1 -- 172.21.15.57:0/1300764540 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fc39c003b40 con 0x7fc3ac106070 2023-07-20T18:52:10.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.373+0000 7fc39b7fe700 1 -- 172.21.15.57:0/1300764540 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc39c024710 con 0x7fc3ac106070 2023-07-20T18:52:10.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.373+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/1300764540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ac19c630 con 0x7fc3ac106070 2023-07-20T18:52:10.079 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.373+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/1300764540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ac19ca50 con 0x7fc3ac106070 2023-07-20T18:52:10.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.374+0000 7fc39b7fe700 1 -- 172.21.15.57:0/1300764540 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc39c01ab30 con 0x7fc3ac106070 2023-07-20T18:52:10.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.374+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/1300764540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ac061da0 con 0x7fc3ac106070 2023-07-20T18:52:10.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.375+0000 7fc39b7fe700 1 --2- 172.21.15.57:0/1300764540 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc3940388c0 0x7fc39403ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:10.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.375+0000 7fc39b7fe700 1 -- 172.21.15.57:0/1300764540 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc39c04ca10 con 0x7fc3ac106070 2023-07-20T18:52:10.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.375+0000 7fc3a9d9b700 1 --2- 172.21.15.57:0/1300764540 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc3940388c0 0x7fc39403ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:10.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.376+0000 7fc3a9d9b700 1 --2- 172.21.15.57:0/1300764540 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc3940388c0 0x7fc39403ad80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0006fd0 tx=0x7fc3a0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:10.080 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.380+0000 7fc39b7fe700 1 -- 172.21.15.57:0/1300764540 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc39c01a950 con 0x7fc3ac106070 2023-07-20T18:52:10.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.583+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/1300764540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fc3ac19f8b0 con 0x7fc3ac106070 2023-07-20T18:52:10.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.584+0000 7fc39b7fe700 1 -- 172.21.15.57:0/1300764540 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fc39c018350 con 0x7fc3ac106070 2023-07-20T18:52:10.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.586+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/1300764540 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc3940388c0 msgr2=0x7fc39403ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:10.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.586+0000 7fc3b0cc1700 1 --2- 172.21.15.57:0/1300764540 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc3940388c0 0x7fc39403ad80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0006fd0 tx=0x7fc3a0006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:10.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.586+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/1300764540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 msgr2=0x7fc3ac19bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:10.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.586+0000 7fc3b0cc1700 1 --2- 172.21.15.57:0/1300764540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac19bef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc39c003f20 tx=0x7fc39c011b00 comp rx=0 tx=0).stop 2023-07-20T18:52:10.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.586+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/1300764540 shutdown_connections 2023-07-20T18:52:10.081 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.586+0000 7fc3b0cc1700 1 --2- 172.21.15.57:0/1300764540 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc3940388c0 0x7fc39403ad80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:10.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.586+0000 7fc3b0cc1700 1 --2- 172.21.15.57:0/1300764540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac106070 0x7fc3ac19bef0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:10.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.586+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/1300764540 >> 172.21.15.57:0/1300764540 conn(0x7fc3ac101610 msgr2=0x7fc3ac102380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:10.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.587+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/1300764540 shutdown_connections 2023-07-20T18:52:10.082 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:09.587+0000 7fc3b0cc1700 1 -- 172.21.15.57:0/1300764540 wait complete. 2023-07-20T18:52:11.356 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.589+0000 7f29b7b0b700 1 Processor -- start 2023-07-20T18:52:11.357 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.589+0000 7f29b7b0b700 1 -- start start 2023-07-20T18:52:11.357 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.589+0000 7f29b7b0b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b007a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:11.357 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.589+0000 7f29b7b0b700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b007a8b0 con 0x7f29b007be80 2023-07-20T18:52:11.357 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.589+0000 7f29b58a7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b007a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:11.357 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.589+0000 7f29b58a7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b007a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41008/0 (socket says 172.21.15.57:41008) 2023-07-20T18:52:11.357 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.589+0000 7f29b58a7700 1 -- 172.21.15.57:0/3977151454 learned_addr learned my addr 172.21.15.57:0/3977151454 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:11.357 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.590+0000 7f29b58a7700 1 -- 172.21.15.57:0/3977151454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b007a9f0 con 0x7f29b007be80 2023-07-20T18:52:11.358 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.590+0000 7f29b58a7700 1 --2- 172.21.15.57:0/3977151454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b007a2e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0098a0 tx=0x7f29ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4476e96423ceefbb server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:11.358 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.590+0000 7f29b48a5700 1 -- 172.21.15.57:0/3977151454 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29ac004160 con 0x7f29b007be80 2023-07-20T18:52:11.358 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.590+0000 7f29b48a5700 1 -- 172.21.15.57:0/3977151454 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f29ac0042c0 con 0x7f29b007be80 2023-07-20T18:52:11.358 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.590+0000 7f29b48a5700 1 -- 172.21.15.57:0/3977151454 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29ac011a80 con 0x7f29b007be80 2023-07-20T18:52:11.358 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.591+0000 7f29b7b0b700 1 -- 172.21.15.57:0/3977151454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 msgr2=0x7f29b007a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:11.358 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.591+0000 7f29b7b0b700 1 --2- 172.21.15.57:0/3977151454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b007a2e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0098a0 tx=0x7f29ac009bb0 comp rx=0 tx=0).stop 2023-07-20T18:52:11.359 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.591+0000 7f29b7b0b700 1 -- 172.21.15.57:0/3977151454 shutdown_connections 2023-07-20T18:52:11.359 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.591+0000 7f29b7b0b700 1 --2- 172.21.15.57:0/3977151454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b007a2e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:11.359 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.591+0000 7f29b7b0b700 1 -- 172.21.15.57:0/3977151454 >> 172.21.15.57:0/3977151454 conn(0x7f29b0101610 msgr2=0x7f29b0103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:11.359 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.591+0000 7f29b7b0b700 1 -- 172.21.15.57:0/3977151454 shutdown_connections 2023-07-20T18:52:11.359 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.591+0000 7f29b7b0b700 1 -- 172.21.15.57:0/3977151454 wait complete. 2023-07-20T18:52:11.359 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.592+0000 7f29b7b0b700 1 Processor -- start 2023-07-20T18:52:11.359 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.592+0000 7f29b7b0b700 1 -- start start 2023-07-20T18:52:11.360 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.593+0000 7f29b7b0b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b019be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:11.360 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.593+0000 7f29b7b0b700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b019c3b0 con 0x7f29b007be80 2023-07-20T18:52:11.360 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.593+0000 7f29b58a7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b019be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:11.360 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.593+0000 7f29b58a7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b019be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41020/0 (socket says 172.21.15.57:41020) 2023-07-20T18:52:11.360 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.593+0000 7f29b58a7700 1 -- 172.21.15.57:0/1039457344 learned_addr learned my addr 172.21.15.57:0/1039457344 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:11.360 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.593+0000 7f29b58a7700 1 -- 172.21.15.57:0/1039457344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac009550 con 0x7f29b007be80 2023-07-20T18:52:11.360 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.593+0000 7f29b58a7700 1 --2- 172.21.15.57:0/1039457344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b019be70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f29ac003f20 tx=0x7f29ac011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:11.360 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.594+0000 7f29a6ffd700 1 -- 172.21.15.57:0/1039457344 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29ac003b20 con 0x7f29b007be80 2023-07-20T18:52:11.361 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.594+0000 7f29a6ffd700 1 -- 172.21.15.57:0/1039457344 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f29ac01a910 con 0x7f29b007be80 2023-07-20T18:52:11.361 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.594+0000 7f29b7b0b700 1 -- 172.21.15.57:0/1039457344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b019c5b0 con 0x7f29b007be80 2023-07-20T18:52:11.361 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.594+0000 7f29a6ffd700 1 -- 172.21.15.57:0/1039457344 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29ac0248c0 con 0x7f29b007be80 2023-07-20T18:52:11.361 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.594+0000 7f29b7b0b700 1 -- 172.21.15.57:0/1039457344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b019c9d0 con 0x7f29b007be80 2023-07-20T18:52:11.361 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.595+0000 7f29a6ffd700 1 -- 172.21.15.57:0/1039457344 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f29ac01aa80 con 0x7f29b007be80 2023-07-20T18:52:11.361 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.595+0000 7f29b7b0b700 1 -- 172.21.15.57:0/1039457344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b0061da0 con 0x7f29b007be80 2023-07-20T18:52:11.361 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.596+0000 7f29a6ffd700 1 --2- 172.21.15.57:0/1039457344 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f299c038910 0x7f299c03add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:11.362 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.596+0000 7f29a6ffd700 1 -- 172.21.15.57:0/1039457344 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f29ac04cc40 con 0x7f29b007be80 2023-07-20T18:52:11.362 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.596+0000 7f29b50a6700 1 --2- 172.21.15.57:0/1039457344 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f299c038910 0x7f299c03add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:11.362 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.596+0000 7f29b50a6700 1 --2- 172.21.15.57:0/1039457344 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f299c038910 0x7f299c03add0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f29a0006fd0 tx=0x7f29a0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:11.362 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.600+0000 7f29a6ffd700 1 -- 172.21.15.57:0/1039457344 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f29ac01adf0 con 0x7f29b007be80 2023-07-20T18:52:11.362 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.918+0000 7f29b7b0b700 1 -- 172.21.15.57:0/1039457344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f29b019f830 con 0x7f29b007be80 2023-07-20T18:52:11.362 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.921+0000 7f29a6ffd700 1 -- 172.21.15.57:0/1039457344 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f29ac018b40 con 0x7f29b007be80 2023-07-20T18:52:11.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.923+0000 7f29b7b0b700 1 -- 172.21.15.57:0/1039457344 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f299c038910 msgr2=0x7f299c03add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:11.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.923+0000 7f29b7b0b700 1 --2- 172.21.15.57:0/1039457344 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f299c038910 0x7f299c03add0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f29a0006fd0 tx=0x7f29a0006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:11.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.923+0000 7f29b7b0b700 1 -- 172.21.15.57:0/1039457344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 msgr2=0x7f29b019be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:11.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.923+0000 7f29b7b0b700 1 --2- 172.21.15.57:0/1039457344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b019be70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f29ac003f20 tx=0x7f29ac011b00 comp rx=0 tx=0).stop 2023-07-20T18:52:11.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.924+0000 7f29b7b0b700 1 -- 172.21.15.57:0/1039457344 shutdown_connections 2023-07-20T18:52:11.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.924+0000 7f29b7b0b700 1 --2- 172.21.15.57:0/1039457344 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f299c038910 0x7f299c03add0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:11.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.924+0000 7f29b7b0b700 1 --2- 172.21.15.57:0/1039457344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29b007be80 0x7f29b019be70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:11.363 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.924+0000 7f29b7b0b700 1 -- 172.21.15.57:0/1039457344 >> 172.21.15.57:0/1039457344 conn(0x7f29b0101610 msgr2=0x7f29b01022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:11.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.924+0000 7f29b7b0b700 1 -- 172.21.15.57:0/1039457344 shutdown_connections 2023-07-20T18:52:11.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2023-07-20T18:52:10.924+0000 7f29b7b0b700 1 -- 172.21.15.57:0/1039457344 wait complete. 2023-07-20T18:52:11.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-20T18:52:11.364 INFO:teuthology.orchestra.run.smithi057.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-20T18:52:11.364 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:11.364 INFO:teuthology.orchestra.run.smithi057.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-20T18:52:11.364 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:11.365 INFO:teuthology.orchestra.run.smithi057.stdout:Or, if you are only running a single cluster on this host: 2023-07-20T18:52:11.365 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:11.365 INFO:teuthology.orchestra.run.smithi057.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-20T18:52:11.365 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:11.365 INFO:teuthology.orchestra.run.smithi057.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-20T18:52:11.365 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:11.365 INFO:teuthology.orchestra.run.smithi057.stdout: ceph telemetry on 2023-07-20T18:52:11.366 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:11.366 INFO:teuthology.orchestra.run.smithi057.stdout:For more information see: 2023-07-20T18:52:11.366 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:11.366 INFO:teuthology.orchestra.run.smithi057.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-20T18:52:11.366 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:11.366 INFO:teuthology.orchestra.run.smithi057.stdout:Bootstrap complete. 2023-07-20T18:52:11.416 INFO:tasks.cephadm:Fetching config... 2023-07-20T18:52:11.416 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:52:11.416 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-20T18:52:11.462 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-20T18:52:11.462 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:52:11.462 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-20T18:52:11.551 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-20T18:52:11.551 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:52:11.552 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-20T18:52:11.662 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-20T18:52:11.663 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:52:11.663 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-20T18:52:11.745 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-20T18:52:11.746 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6TRNby6luT5sHlY1YKnUvVFG2ORB3FJ1J14fQejW9LpsMw4qjVkPA0pJjKEWfjqDUTI8jZb6M9/M59LcuelRsPOAhogtK5JGkXiVFsI2EgxIh59CCX5UwHVXjs6F2yrogEJv5paJr8iCAvv/XINlv2vKnfw5Jpvlhd+MFJ7FC+tiZfrRK2M1EBv2MKdqsA8fZNoR2R6mrAqlksdz2DCTwCCUAL8WKsAIcUcZO9Dvmm5oJn7+W9aT5avy/btj+y2fEX/+wcnqbAUJ3kZJfkQIOWKxDMxVjH1QxZrDhiY7Vk5Frq61ADKWc9uEDtQkq96YlQr9Yr0CZcv3FDwLz80GXnn0e7ZVm87dD25Pzrd825YBh3Vptfmv0GaIBwrBsSTvG0SjOwTlDsVWht9NdtkSDA+lj698a2BKKR8Y0xVVsoYNh7Y8S2tTjuiNN3RCaMmmg9T965hcRcAZARpY5N9JLfVprRh1bVkM4gwQBHbIcnJ+0YRQXljsN/zIxVHC5d+U= ceph-38a58c3c-272e-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T18:52:11.887 INFO:teuthology.orchestra.run.smithi057.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6TRNby6luT5sHlY1YKnUvVFG2ORB3FJ1J14fQejW9LpsMw4qjVkPA0pJjKEWfjqDUTI8jZb6M9/M59LcuelRsPOAhogtK5JGkXiVFsI2EgxIh59CCX5UwHVXjs6F2yrogEJv5paJr8iCAvv/XINlv2vKnfw5Jpvlhd+MFJ7FC+tiZfrRK2M1EBv2MKdqsA8fZNoR2R6mrAqlksdz2DCTwCCUAL8WKsAIcUcZO9Dvmm5oJn7+W9aT5avy/btj+y2fEX/+wcnqbAUJ3kZJfkQIOWKxDMxVjH1QxZrDhiY7Vk5Frq61ADKWc9uEDtQkq96YlQr9Yr0CZcv3FDwLz80GXnn0e7ZVm87dD25Pzrd825YBh3Vptfmv0GaIBwrBsSTvG0SjOwTlDsVWht9NdtkSDA+lj698a2BKKR8Y0xVVsoYNh7Y8S2tTjuiNN3RCaMmmg9T965hcRcAZARpY5N9JLfVprRh1bVkM4gwQBHbIcnJ+0YRQXljsN/zIxVHC5d+U= ceph-38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:52:11.920 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6TRNby6luT5sHlY1YKnUvVFG2ORB3FJ1J14fQejW9LpsMw4qjVkPA0pJjKEWfjqDUTI8jZb6M9/M59LcuelRsPOAhogtK5JGkXiVFsI2EgxIh59CCX5UwHVXjs6F2yrogEJv5paJr8iCAvv/XINlv2vKnfw5Jpvlhd+MFJ7FC+tiZfrRK2M1EBv2MKdqsA8fZNoR2R6mrAqlksdz2DCTwCCUAL8WKsAIcUcZO9Dvmm5oJn7+W9aT5avy/btj+y2fEX/+wcnqbAUJ3kZJfkQIOWKxDMxVjH1QxZrDhiY7Vk5Frq61ADKWc9uEDtQkq96YlQr9Yr0CZcv3FDwLz80GXnn0e7ZVm87dD25Pzrd825YBh3Vptfmv0GaIBwrBsSTvG0SjOwTlDsVWht9NdtkSDA+lj698a2BKKR8Y0xVVsoYNh7Y8S2tTjuiNN3RCaMmmg9T965hcRcAZARpY5N9JLfVprRh1bVkM4gwQBHbIcnJ+0YRQXljsN/zIxVHC5d+U= ceph-38a58c3c-272e-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T18:52:11.966 INFO:teuthology.orchestra.run.smithi088.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6TRNby6luT5sHlY1YKnUvVFG2ORB3FJ1J14fQejW9LpsMw4qjVkPA0pJjKEWfjqDUTI8jZb6M9/M59LcuelRsPOAhogtK5JGkXiVFsI2EgxIh59CCX5UwHVXjs6F2yrogEJv5paJr8iCAvv/XINlv2vKnfw5Jpvlhd+MFJ7FC+tiZfrRK2M1EBv2MKdqsA8fZNoR2R6mrAqlksdz2DCTwCCUAL8WKsAIcUcZO9Dvmm5oJn7+W9aT5avy/btj+y2fEX/+wcnqbAUJ3kZJfkQIOWKxDMxVjH1QxZrDhiY7Vk5Frq61ADKWc9uEDtQkq96YlQr9Yr0CZcv3FDwLz80GXnn0e7ZVm87dD25Pzrd825YBh3Vptfmv0GaIBwrBsSTvG0SjOwTlDsVWht9NdtkSDA+lj698a2BKKR8Y0xVVsoYNh7Y8S2tTjuiNN3RCaMmmg9T965hcRcAZARpY5N9JLfVprRh1bVkM4gwQBHbIcnJ+0YRQXljsN/zIxVHC5d+U= ceph-38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:52:11.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6TRNby6luT5sHlY1YKnUvVFG2ORB3FJ1J14fQejW9LpsMw4qjVkPA0pJjKEWfjqDUTI8jZb6M9/M59LcuelRsPOAhogtK5JGkXiVFsI2EgxIh59CCX5UwHVXjs6F2yrogEJv5paJr8iCAvv/XINlv2vKnfw5Jpvlhd+MFJ7FC+tiZfrRK2M1EBv2MKdqsA8fZNoR2R6mrAqlksdz2DCTwCCUAL8WKsAIcUcZO9Dvmm5oJn7+W9aT5avy/btj+y2fEX/+wcnqbAUJ3kZJfkQIOWKxDMxVjH1QxZrDhiY7Vk5Frq61ADKWc9uEDtQkq96YlQr9Yr0CZcv3FDwLz80GXnn0e7ZVm87dD25Pzrd825YBh3Vptfmv0GaIBwrBsSTvG0SjOwTlDsVWht9NdtkSDA+lj698a2BKKR8Y0xVVsoYNh7Y8S2tTjuiNN3RCaMmmg9T965hcRcAZARpY5N9JLfVprRh1bVkM4gwQBHbIcnJ+0YRQXljsN/zIxVHC5d+U= ceph-38a58c3c-272e-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T18:52:12.024 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6TRNby6luT5sHlY1YKnUvVFG2ORB3FJ1J14fQejW9LpsMw4qjVkPA0pJjKEWfjqDUTI8jZb6M9/M59LcuelRsPOAhogtK5JGkXiVFsI2EgxIh59CCX5UwHVXjs6F2yrogEJv5paJr8iCAvv/XINlv2vKnfw5Jpvlhd+MFJ7FC+tiZfrRK2M1EBv2MKdqsA8fZNoR2R6mrAqlksdz2DCTwCCUAL8WKsAIcUcZO9Dvmm5oJn7+W9aT5avy/btj+y2fEX/+wcnqbAUJ3kZJfkQIOWKxDMxVjH1QxZrDhiY7Vk5Frq61ADKWc9uEDtQkq96YlQr9Yr0CZcv3FDwLz80GXnn0e7ZVm87dD25Pzrd825YBh3Vptfmv0GaIBwrBsSTvG0SjOwTlDsVWht9NdtkSDA+lj698a2BKKR8Y0xVVsoYNh7Y8S2tTjuiNN3RCaMmmg9T965hcRcAZARpY5N9JLfVprRh1bVkM4gwQBHbIcnJ+0YRQXljsN/zIxVHC5d+U= ceph-38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:52:12.037 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-20T18:52:12.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:11 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1039457344' entity='client.admin' 2023-07-20T18:52:13.081 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:52:14.318 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:14.318 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:14.318 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi057", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:52:14.318 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:14.319 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:14.319 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:15.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.228+0000 7fec9aef4700 1 -- 172.21.15.57:0/2374697853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec94100280 msgr2=0x7fec941006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:15.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.228+0000 7fec9aef4700 1 --2- 172.21.15.57:0/2374697853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec94100280 0x7fec941006a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec88009910 tx=0x7fec88009c20 comp rx=0 tx=0).stop 2023-07-20T18:52:15.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.228+0000 7fec9aef4700 1 -- 172.21.15.57:0/2374697853 shutdown_connections 2023-07-20T18:52:15.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.228+0000 7fec9aef4700 1 --2- 172.21.15.57:0/2374697853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec94100280 0x7fec941006a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:15.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.228+0000 7fec9aef4700 1 -- 172.21.15.57:0/2374697853 >> 172.21.15.57:0/2374697853 conn(0x7fec940fb800 msgr2=0x7fec940fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:15.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.228+0000 7fec9aef4700 1 -- 172.21.15.57:0/2374697853 shutdown_connections 2023-07-20T18:52:15.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.228+0000 7fec9aef4700 1 -- 172.21.15.57:0/2374697853 wait complete. 2023-07-20T18:52:15.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.229+0000 7fec9aef4700 1 Processor -- start 2023-07-20T18:52:15.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.229+0000 7fec9aef4700 1 -- start start 2023-07-20T18:52:15.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.230+0000 7fec9aef4700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec94100280 0x7fec94195430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:15.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.230+0000 7fec9aef4700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec94195970 con 0x7fec94100280 2023-07-20T18:52:15.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.230+0000 7fec98c90700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec94100280 0x7fec94195430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:15.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.230+0000 7fec98c90700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec94100280 0x7fec94195430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36100/0 (socket says 172.21.15.57:36100) 2023-07-20T18:52:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.230+0000 7fec98c90700 1 -- 172.21.15.57:0/139806954 learned_addr learned my addr 172.21.15.57:0/139806954 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.230+0000 7fec98c90700 1 -- 172.21.15.57:0/139806954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec880095f0 con 0x7fec94100280 2023-07-20T18:52:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.231+0000 7fec98c90700 1 --2- 172.21.15.57:0/139806954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec94100280 0x7fec94195430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec88006010 tx=0x7fec88004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.231+0000 7fec91ffb700 1 -- 172.21.15.57:0/139806954 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec8801d070 con 0x7fec94100280 2023-07-20T18:52:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.231+0000 7fec9aef4700 1 -- 172.21.15.57:0/139806954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec94195b70 con 0x7fec94100280 2023-07-20T18:52:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.231+0000 7fec91ffb700 1 -- 172.21.15.57:0/139806954 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fec88004500 con 0x7fec94100280 2023-07-20T18:52:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.231+0000 7fec91ffb700 1 -- 172.21.15.57:0/139806954 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec88003b10 con 0x7fec94100280 2023-07-20T18:52:15.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.231+0000 7fec9aef4700 1 -- 172.21.15.57:0/139806954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec94195fb0 con 0x7fec94100280 2023-07-20T18:52:15.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.232+0000 7fec91ffb700 1 -- 172.21.15.57:0/139806954 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fec8800bc20 con 0x7fec94100280 2023-07-20T18:52:15.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.232+0000 7fec9aef4700 1 -- 172.21.15.57:0/139806954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9418eed0 con 0x7fec94100280 2023-07-20T18:52:15.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.233+0000 7fec91ffb700 1 --2- 172.21.15.57:0/139806954 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fec840387b0 0x7fec8403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:15.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.233+0000 7fec91ffb700 1 -- 172.21.15.57:0/139806954 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fec8804c8f0 con 0x7fec94100280 2023-07-20T18:52:15.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.233+0000 7fec93fff700 1 --2- 172.21.15.57:0/139806954 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fec840387b0 0x7fec8403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:15.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.233+0000 7fec93fff700 1 --2- 172.21.15.57:0/139806954 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fec840387b0 0x7fec8403ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fec80006fd0 tx=0x7fec80006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:15.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.238+0000 7fec91ffb700 1 -- 172.21.15.57:0/139806954 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fec880042f0 con 0x7fec94100280 2023-07-20T18:52:15.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.448+0000 7fec9aef4700 1 -- 172.21.15.57:0/139806954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fec9402c970 con 0x7fec94100280 2023-07-20T18:52:15.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.452+0000 7fec91ffb700 1 -- 172.21.15.57:0/139806954 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fec88005c00 con 0x7fec94100280 2023-07-20T18:52:15.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.455+0000 7fec9aef4700 1 -- 172.21.15.57:0/139806954 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fec840387b0 msgr2=0x7fec8403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:15.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.455+0000 7fec9aef4700 1 --2- 172.21.15.57:0/139806954 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fec840387b0 0x7fec8403ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fec80006fd0 tx=0x7fec80006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:15.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.455+0000 7fec9aef4700 1 -- 172.21.15.57:0/139806954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec94100280 msgr2=0x7fec94195430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:15.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.455+0000 7fec9aef4700 1 --2- 172.21.15.57:0/139806954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec94100280 0x7fec94195430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec88006010 tx=0x7fec88004ff0 comp rx=0 tx=0).stop 2023-07-20T18:52:15.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.455+0000 7fec9aef4700 1 -- 172.21.15.57:0/139806954 shutdown_connections 2023-07-20T18:52:15.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.455+0000 7fec9aef4700 1 --2- 172.21.15.57:0/139806954 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fec840387b0 0x7fec8403ac70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:15.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.455+0000 7fec9aef4700 1 --2- 172.21.15.57:0/139806954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec94100280 0x7fec94195430 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:15.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.455+0000 7fec9aef4700 1 -- 172.21.15.57:0/139806954 >> 172.21.15.57:0/139806954 conn(0x7fec940fb800 msgr2=0x7fec940fc520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.455+0000 7fec9aef4700 1 -- 172.21.15.57:0/139806954 shutdown_connections 2023-07-20T18:52:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:15.455+0000 7fec9aef4700 1 -- 172.21.15.57:0/139806954 wait complete. 2023-07-20T18:52:16.081 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-20T18:52:16.082 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-20T18:52:16.812 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:52:16.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:16 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/139806954' entity='client.admin' 2023-07-20T18:52:16.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:16 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:16.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:16 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:18.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.269+0000 7f344cc75700 1 -- 172.21.15.57:0/2607105876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3448100280 msgr2=0x7f34481006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:18.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.269+0000 7f344cc75700 1 --2- 172.21.15.57:0/2607105876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3448100280 0x7f34481006a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3438009910 tx=0x7f3438009c20 comp rx=0 tx=0).stop 2023-07-20T18:52:18.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.269+0000 7f344cc75700 1 -- 172.21.15.57:0/2607105876 shutdown_connections 2023-07-20T18:52:18.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.269+0000 7f344cc75700 1 --2- 172.21.15.57:0/2607105876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3448100280 0x7f34481006a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:18.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.269+0000 7f344cc75700 1 -- 172.21.15.57:0/2607105876 >> 172.21.15.57:0/2607105876 conn(0x7f34480fb800 msgr2=0x7f34480fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:18.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.269+0000 7f344cc75700 1 -- 172.21.15.57:0/2607105876 shutdown_connections 2023-07-20T18:52:18.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.269+0000 7f344cc75700 1 -- 172.21.15.57:0/2607105876 wait complete. 2023-07-20T18:52:18.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.271+0000 7f344cc75700 1 Processor -- start 2023-07-20T18:52:18.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.271+0000 7f344cc75700 1 -- start start 2023-07-20T18:52:18.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.271+0000 7f344cc75700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3448100280 0x7f3448197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:18.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.271+0000 7f344cc75700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3448197b70 con 0x7f3448100280 2023-07-20T18:52:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.271+0000 7f344659c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3448100280 0x7f3448197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.271+0000 7f344659c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3448100280 0x7f3448197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36116/0 (socket says 172.21.15.57:36116) 2023-07-20T18:52:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.271+0000 7f344659c700 1 -- 172.21.15.57:0/484940062 learned_addr learned my addr 172.21.15.57:0/484940062 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.272+0000 7f344659c700 1 -- 172.21.15.57:0/484940062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34380095f0 con 0x7f3448100280 2023-07-20T18:52:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.272+0000 7f344659c700 1 --2- 172.21.15.57:0/484940062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3448100280 0x7f3448197630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3438006010 tx=0x7f3438004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.272+0000 7f34377fe700 1 -- 172.21.15.57:0/484940062 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f343801d070 con 0x7f3448100280 2023-07-20T18:52:18.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.272+0000 7f34377fe700 1 -- 172.21.15.57:0/484940062 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f343800bc30 con 0x7f3448100280 2023-07-20T18:52:18.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.272+0000 7f34377fe700 1 -- 172.21.15.57:0/484940062 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3438003aa0 con 0x7f3448100280 2023-07-20T18:52:18.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.272+0000 7f344cc75700 1 -- 172.21.15.57:0/484940062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3448197d70 con 0x7f3448100280 2023-07-20T18:52:18.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.272+0000 7f344cc75700 1 -- 172.21.15.57:0/484940062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3448198210 con 0x7f3448100280 2023-07-20T18:52:18.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.273+0000 7f34377fe700 1 -- 172.21.15.57:0/484940062 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f3438003d10 con 0x7f3448100280 2023-07-20T18:52:18.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.273+0000 7f344cc75700 1 -- 172.21.15.57:0/484940062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34481912d0 con 0x7f3448100280 2023-07-20T18:52:18.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.274+0000 7f34377fe700 1 --2- 172.21.15.57:0/484940062 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3430038800 0x7f343003acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:18.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.274+0000 7f34377fe700 1 -- 172.21.15.57:0/484940062 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f343804c4b0 con 0x7f3448100280 2023-07-20T18:52:18.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.274+0000 7f3445d9b700 1 --2- 172.21.15.57:0/484940062 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3430038800 0x7f343003acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:18.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.275+0000 7f3445d9b700 1 --2- 172.21.15.57:0/484940062 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3430038800 0x7f343003acc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f343c006fd0 tx=0x7f343c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:18.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.279+0000 7f34377fe700 1 -- 172.21.15.57:0/484940062 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f343802abb0 con 0x7f3448100280 2023-07-20T18:52:18.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.492+0000 7f344cc75700 1 -- 172.21.15.57:0/484940062 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f3448060c40 con 0x7f3430038800 2023-07-20T18:52:18.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.496+0000 7f34377fe700 1 -- 172.21.15.57:0/484940062 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3448060c40 con 0x7f3430038800 2023-07-20T18:52:18.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.498+0000 7f344cc75700 1 -- 172.21.15.57:0/484940062 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3430038800 msgr2=0x7f343003acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:18.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.498+0000 7f344cc75700 1 --2- 172.21.15.57:0/484940062 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3430038800 0x7f343003acc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f343c006fd0 tx=0x7f343c006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:18.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.498+0000 7f344cc75700 1 -- 172.21.15.57:0/484940062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3448100280 msgr2=0x7f3448197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:18.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.498+0000 7f344cc75700 1 --2- 172.21.15.57:0/484940062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3448100280 0x7f3448197630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3438006010 tx=0x7f3438004ff0 comp rx=0 tx=0).stop 2023-07-20T18:52:18.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.499+0000 7f344cc75700 1 -- 172.21.15.57:0/484940062 shutdown_connections 2023-07-20T18:52:18.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.499+0000 7f344cc75700 1 --2- 172.21.15.57:0/484940062 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3430038800 0x7f343003acc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:18.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.499+0000 7f344cc75700 1 --2- 172.21.15.57:0/484940062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3448100280 0x7f3448197630 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:18.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.499+0000 7f344cc75700 1 -- 172.21.15.57:0/484940062 >> 172.21.15.57:0/484940062 conn(0x7f34480fb800 msgr2=0x7f34480fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:18.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.499+0000 7f344cc75700 1 -- 172.21.15.57:0/484940062 shutdown_connections 2023-07-20T18:52:18.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:18.499+0000 7f344cc75700 1 -- 172.21.15.57:0/484940062 wait complete. 2023-07-20T18:52:18.991 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi088 2023-07-20T18:52:18.991 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:52:18.992 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/etc/ceph/ceph.conf 2023-07-20T18:52:19.012 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:52:19.012 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:52:19.072 INFO:tasks.cephadm:Adding host smithi088 to orchestrator... 2023-07-20T18:52:19.072 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch host add smithi088 2023-07-20T18:52:19.818 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:52:19.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:19 smithi057 ceph-mon[126340]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:52:19.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:19 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:19.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:19 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:19.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:19 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:19.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:19 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:52:19.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:19 smithi057 ceph-mon[126340]: Updating smithi057:/etc/ceph/ceph.conf 2023-07-20T18:52:20.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:20 smithi057 ceph-mon[126340]: Updating smithi057:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:52:20.724 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:20 smithi057 ceph-mon[126340]: Updating smithi057:/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:52:21.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.392+0000 7fbfc85d7700 1 -- 172.21.15.57:0/2223720504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfc0100460 msgr2=0x7fbfc0100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:21.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.392+0000 7fbfc85d7700 1 --2- 172.21.15.57:0/2223720504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfc0100460 0x7fbfc0100840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc009910 tx=0x7fbfbc009c20 comp rx=0 tx=0).stop 2023-07-20T18:52:21.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.392+0000 7fbfc85d7700 1 -- 172.21.15.57:0/2223720504 shutdown_connections 2023-07-20T18:52:21.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.392+0000 7fbfc85d7700 1 --2- 172.21.15.57:0/2223720504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfc0100460 0x7fbfc0100840 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:21.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.392+0000 7fbfc85d7700 1 -- 172.21.15.57:0/2223720504 >> 172.21.15.57:0/2223720504 conn(0x7fbfc00fbd30 msgr2=0x7fbfc00fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:21.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.392+0000 7fbfc85d7700 1 -- 172.21.15.57:0/2223720504 shutdown_connections 2023-07-20T18:52:21.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.392+0000 7fbfc85d7700 1 -- 172.21.15.57:0/2223720504 wait complete. 2023-07-20T18:52:21.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.393+0000 7fbfc85d7700 1 Processor -- start 2023-07-20T18:52:21.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.394+0000 7fbfc85d7700 1 -- start start 2023-07-20T18:52:21.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.394+0000 7fbfc85d7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfc0100460 0x7fbfc019d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:21.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.394+0000 7fbfc85d7700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc019dd30 con 0x7fbfc0100460 2023-07-20T18:52:21.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.394+0000 7fbfc6373700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfc0100460 0x7fbfc019d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:21.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.394+0000 7fbfc6373700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfc0100460 0x7fbfc019d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36136/0 (socket says 172.21.15.57:36136) 2023-07-20T18:52:21.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.394+0000 7fbfc6373700 1 -- 172.21.15.57:0/502801675 learned_addr learned my addr 172.21.15.57:0/502801675 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:21.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.394+0000 7fbfc6373700 1 -- 172.21.15.57:0/502801675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc0095f0 con 0x7fbfc0100460 2023-07-20T18:52:21.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.395+0000 7fbfc6373700 1 --2- 172.21.15.57:0/502801675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfc0100460 0x7fbfc019d7f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc005f50 tx=0x7fbfbc004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:21.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.395+0000 7fbfb77fe700 1 -- 172.21.15.57:0/502801675 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbfbc01d070 con 0x7fbfc0100460 2023-07-20T18:52:21.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.395+0000 7fbfb77fe700 1 -- 172.21.15.57:0/502801675 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fbfbc00bc30 con 0x7fbfc0100460 2023-07-20T18:52:21.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.395+0000 7fbfc85d7700 1 -- 172.21.15.57:0/502801675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc019df30 con 0x7fbfc0100460 2023-07-20T18:52:21.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.395+0000 7fbfb77fe700 1 -- 172.21.15.57:0/502801675 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbfbc003a40 con 0x7fbfc0100460 2023-07-20T18:52:21.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.395+0000 7fbfc85d7700 1 -- 172.21.15.57:0/502801675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc019e350 con 0x7fbfc0100460 2023-07-20T18:52:21.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.396+0000 7fbfb77fe700 1 -- 172.21.15.57:0/502801675 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fbfbc003ba0 con 0x7fbfc0100460 2023-07-20T18:52:21.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.396+0000 7fbfc85d7700 1 -- 172.21.15.57:0/502801675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc0061da0 con 0x7fbfc0100460 2023-07-20T18:52:21.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.397+0000 7fbfb77fe700 1 --2- 172.21.15.57:0/502801675 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fbfac038800 0x7fbfac03acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:21.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.397+0000 7fbfb77fe700 1 -- 172.21.15.57:0/502801675 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbfbc04c510 con 0x7fbfc0100460 2023-07-20T18:52:21.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.397+0000 7fbfc5b72700 1 --2- 172.21.15.57:0/502801675 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fbfac038800 0x7fbfac03acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:21.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.398+0000 7fbfc5b72700 1 --2- 172.21.15.57:0/502801675 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fbfac038800 0x7fbfac03acc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0006fd0 tx=0x7fbfb0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:21.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.402+0000 7fbfb77fe700 1 -- 172.21.15.57:0/502801675 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbfbc00b6f0 con 0x7fbfc0100460 2023-07-20T18:52:21.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:21.614+0000 7fbfc85d7700 1 -- 172.21.15.57:0/502801675 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi088", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfc0074240 con 0x7fbfac038800 2023-07-20T18:52:21.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:21 smithi057 ceph-mon[126340]: Updating smithi057:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T18:52:21.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:21.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:21.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:22.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:22 smithi057 ceph-mon[126340]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi088", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:52:24.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.980+0000 7fbfb77fe700 1 -- 172.21.15.57:0/502801675 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fbfc0074240 con 0x7fbfac038800 2023-07-20T18:52:24.021 INFO:teuthology.orchestra.run.smithi057.stdout:Added host 'smithi088' with addr '172.21.15.88' 2023-07-20T18:52:24.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.983+0000 7fbfc85d7700 1 -- 172.21.15.57:0/502801675 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fbfac038800 msgr2=0x7fbfac03acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:24.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.983+0000 7fbfc85d7700 1 --2- 172.21.15.57:0/502801675 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fbfac038800 0x7fbfac03acc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0006fd0 tx=0x7fbfb0006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:24.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.983+0000 7fbfc85d7700 1 -- 172.21.15.57:0/502801675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfc0100460 msgr2=0x7fbfc019d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:24.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.983+0000 7fbfc85d7700 1 --2- 172.21.15.57:0/502801675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfc0100460 0x7fbfc019d7f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc005f50 tx=0x7fbfbc004ff0 comp rx=0 tx=0).stop 2023-07-20T18:52:24.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.983+0000 7fbfc85d7700 1 -- 172.21.15.57:0/502801675 shutdown_connections 2023-07-20T18:52:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.983+0000 7fbfc85d7700 1 --2- 172.21.15.57:0/502801675 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fbfac038800 0x7fbfac03acc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.983+0000 7fbfc85d7700 1 --2- 172.21.15.57:0/502801675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfc0100460 0x7fbfc019d7f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.983+0000 7fbfc85d7700 1 -- 172.21.15.57:0/502801675 >> 172.21.15.57:0/502801675 conn(0x7fbfc00fbd30 msgr2=0x7fbfc0103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.983+0000 7fbfc85d7700 1 -- 172.21.15.57:0/502801675 shutdown_connections 2023-07-20T18:52:24.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:23.984+0000 7fbfc85d7700 1 -- 172.21.15.57:0/502801675 wait complete. 2023-07-20T18:52:24.047 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:23 smithi057 ceph-mon[126340]: Deploying cephadm binary to smithi088 2023-07-20T18:52:24.047 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:23 smithi057 ceph-mon[126340]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:24.503 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T18:52:25.229 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:52:25.474 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:24 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:25.474 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:24 smithi057 ceph-mon[126340]: Added host smithi088 2023-07-20T18:52:25.474 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:24 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:25.474 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:24 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:25.475 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:24 smithi057 ceph-mon[126340]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:26.474 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:26 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:26.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.723+0000 7facd89cb700 1 -- 172.21.15.57:0/1895608263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd4100460 msgr2=0x7facd4100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.723+0000 7facd89cb700 1 --2- 172.21.15.57:0/1895608263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd4100460 0x7facd4100840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7facc4009910 tx=0x7facc4009c20 comp rx=0 tx=0).stop 2023-07-20T18:52:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.723+0000 7facd89cb700 1 -- 172.21.15.57:0/1895608263 shutdown_connections 2023-07-20T18:52:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.723+0000 7facd89cb700 1 --2- 172.21.15.57:0/1895608263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd4100460 0x7facd4100840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.723+0000 7facd89cb700 1 -- 172.21.15.57:0/1895608263 >> 172.21.15.57:0/1895608263 conn(0x7facd40fbd30 msgr2=0x7facd40fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.723+0000 7facd89cb700 1 -- 172.21.15.57:0/1895608263 shutdown_connections 2023-07-20T18:52:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.723+0000 7facd89cb700 1 -- 172.21.15.57:0/1895608263 wait complete. 2023-07-20T18:52:26.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.724+0000 7facd89cb700 1 Processor -- start 2023-07-20T18:52:26.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.725+0000 7facd89cb700 1 -- start start 2023-07-20T18:52:26.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.725+0000 7facd89cb700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd4100460 0x7facd419d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:26.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.725+0000 7facd89cb700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd419de00 con 0x7facd4100460 2023-07-20T18:52:26.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.725+0000 7facd259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd4100460 0x7facd419d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.725+0000 7facd259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd4100460 0x7facd419d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33016/0 (socket says 172.21.15.57:33016) 2023-07-20T18:52:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.725+0000 7facd259c700 1 -- 172.21.15.57:0/2261013203 learned_addr learned my addr 172.21.15.57:0/2261013203 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.726+0000 7facd259c700 1 -- 172.21.15.57:0/2261013203 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40095f0 con 0x7facd4100460 2023-07-20T18:52:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.726+0000 7facd259c700 1 --2- 172.21.15.57:0/2261013203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd4100460 0x7facd419d8c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facc4005850 tx=0x7facc4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.726+0000 7facc37fe700 1 -- 172.21.15.57:0/2261013203 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facc401d070 con 0x7facd4100460 2023-07-20T18:52:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.726+0000 7facc37fe700 1 -- 172.21.15.57:0/2261013203 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7facc4004500 con 0x7facd4100460 2023-07-20T18:52:26.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.726+0000 7facc37fe700 1 -- 172.21.15.57:0/2261013203 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facc40039f0 con 0x7facd4100460 2023-07-20T18:52:26.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.726+0000 7facd89cb700 1 -- 172.21.15.57:0/2261013203 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd419e000 con 0x7facd4100460 2023-07-20T18:52:26.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.726+0000 7facd89cb700 1 -- 172.21.15.57:0/2261013203 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd419e3c0 con 0x7facd4100460 2023-07-20T18:52:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.727+0000 7facc37fe700 1 -- 172.21.15.57:0/2261013203 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7facc400b960 con 0x7facd4100460 2023-07-20T18:52:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.727+0000 7facd89cb700 1 -- 172.21.15.57:0/2261013203 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd4061da0 con 0x7facd4100460 2023-07-20T18:52:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.728+0000 7facc37fe700 1 --2- 172.21.15.57:0/2261013203 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7facbc0387b0 0x7facbc03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.728+0000 7facc37fe700 1 -- 172.21.15.57:0/2261013203 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7facc404c940 con 0x7facd4100460 2023-07-20T18:52:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.728+0000 7facd1d9b700 1 --2- 172.21.15.57:0/2261013203 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7facbc0387b0 0x7facbc03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.729+0000 7facd1d9b700 1 --2- 172.21.15.57:0/2261013203 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7facbc0387b0 0x7facbc03ac70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7facc8006fd0 tx=0x7facc8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:26.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.733+0000 7facc37fe700 1 -- 172.21.15.57:0/2261013203 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7facc402a540 con 0x7facd4100460 2023-07-20T18:52:26.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.943+0000 7facd89cb700 1 -- 172.21.15.57:0/2261013203 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7facd4074240 con 0x7facbc0387b0 2023-07-20T18:52:26.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.944+0000 7facc37fe700 1 -- 172.21.15.57:0/2261013203 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7facd4074240 con 0x7facbc0387b0 2023-07-20T18:52:26.984 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:26.984 INFO:teuthology.orchestra.run.smithi057.stdout:[{"addr": "172.21.15.57", "hostname": "smithi057", "labels": [], "status": ""}, {"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}] 2023-07-20T18:52:26.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.947+0000 7facd89cb700 1 -- 172.21.15.57:0/2261013203 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7facbc0387b0 msgr2=0x7facbc03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:26.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.947+0000 7facd89cb700 1 --2- 172.21.15.57:0/2261013203 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7facbc0387b0 0x7facbc03ac70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7facc8006fd0 tx=0x7facc8006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:26.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.947+0000 7facd89cb700 1 -- 172.21.15.57:0/2261013203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd4100460 msgr2=0x7facd419d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:26.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.947+0000 7facd89cb700 1 --2- 172.21.15.57:0/2261013203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd4100460 0x7facd419d8c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facc4005850 tx=0x7facc4004ff0 comp rx=0 tx=0).stop 2023-07-20T18:52:26.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.947+0000 7facd89cb700 1 -- 172.21.15.57:0/2261013203 shutdown_connections 2023-07-20T18:52:26.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.947+0000 7facd89cb700 1 --2- 172.21.15.57:0/2261013203 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7facbc0387b0 0x7facbc03ac70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:26.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.947+0000 7facd89cb700 1 --2- 172.21.15.57:0/2261013203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd4100460 0x7facd419d8c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:26.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.947+0000 7facd89cb700 1 -- 172.21.15.57:0/2261013203 >> 172.21.15.57:0/2261013203 conn(0x7facd40fbd30 msgr2=0x7facd4103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:26.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.947+0000 7facd89cb700 1 -- 172.21.15.57:0/2261013203 shutdown_connections 2023-07-20T18:52:26.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:26.947+0000 7facd89cb700 1 -- 172.21.15.57:0/2261013203 wait complete. 2023-07-20T18:52:27.506 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2023-07-20T18:52:27.507 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:52:27.508 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2023-07-20T18:52:27.528 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:52:27.529 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:52:27.587 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2023-07-20T18:52:27.587 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch host add smithi192 2023-07-20T18:52:27.724 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:27.724 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:27 smithi057 ceph-mon[126340]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:28.315 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:52:28.922 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:28 smithi057 ceph-mon[126340]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T18:52:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.784+0000 7f536a893700 1 -- 172.21.15.57:0/1394965219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5364108480 msgr2=0x7f5364108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.784+0000 7f536a893700 1 --2- 172.21.15.57:0/1394965219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5364108480 0x7f5364108860 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5354009920 tx=0x7f5354009c30 comp rx=0 tx=0).stop 2023-07-20T18:52:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.784+0000 7f536a893700 1 -- 172.21.15.57:0/1394965219 shutdown_connections 2023-07-20T18:52:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.784+0000 7f536a893700 1 --2- 172.21.15.57:0/1394965219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5364108480 0x7f5364108860 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.784+0000 7f536a893700 1 -- 172.21.15.57:0/1394965219 >> 172.21.15.57:0/1394965219 conn(0x7f53640fdea0 msgr2=0x7f53641002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.784+0000 7f536a893700 1 -- 172.21.15.57:0/1394965219 shutdown_connections 2023-07-20T18:52:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.784+0000 7f536a893700 1 -- 172.21.15.57:0/1394965219 wait complete. 2023-07-20T18:52:29.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.785+0000 7f536a893700 1 Processor -- start 2023-07-20T18:52:29.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.786+0000 7f536a893700 1 -- start start 2023-07-20T18:52:29.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.786+0000 7f536a893700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5364108480 0x7f5364074c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:29.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.786+0000 7f536a893700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5364078c70 con 0x7f5364108480 2023-07-20T18:52:29.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.786+0000 7f5363fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5364108480 0x7f5364074c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:29.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.786+0000 7f5363fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5364108480 0x7f5364074c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33026/0 (socket says 172.21.15.57:33026) 2023-07-20T18:52:29.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.786+0000 7f5363fff700 1 -- 172.21.15.57:0/520011969 learned_addr learned my addr 172.21.15.57:0/520011969 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:29.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.787+0000 7f5363fff700 1 -- 172.21.15.57:0/520011969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5354009580 con 0x7f5364108480 2023-07-20T18:52:29.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.787+0000 7f5363fff700 1 --2- 172.21.15.57:0/520011969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5364108480 0x7f5364074c80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f535400b5c0 tx=0x7f5354005010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:29.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.787+0000 7f53617fa700 1 -- 172.21.15.57:0/520011969 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f535401d070 con 0x7f5364108480 2023-07-20T18:52:29.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.787+0000 7f53617fa700 1 -- 172.21.15.57:0/520011969 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f535400bbd0 con 0x7f5364108480 2023-07-20T18:52:29.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.787+0000 7f536a893700 1 -- 172.21.15.57:0/520011969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53640751c0 con 0x7f5364108480 2023-07-20T18:52:29.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.787+0000 7f53617fa700 1 -- 172.21.15.57:0/520011969 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5354003a10 con 0x7f5364108480 2023-07-20T18:52:29.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.787+0000 7f536a893700 1 -- 172.21.15.57:0/520011969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5364075640 con 0x7f5364108480 2023-07-20T18:52:29.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.788+0000 7f53617fa700 1 -- 172.21.15.57:0/520011969 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f535400bd40 con 0x7f5364108480 2023-07-20T18:52:29.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.789+0000 7f536a893700 1 -- 172.21.15.57:0/520011969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5364106730 con 0x7f5364108480 2023-07-20T18:52:29.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.789+0000 7f53617fa700 1 --2- 172.21.15.57:0/520011969 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f534c038850 0x7f534c03ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:29.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.789+0000 7f53617fa700 1 -- 172.21.15.57:0/520011969 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f535404c320 con 0x7f5364108480 2023-07-20T18:52:29.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.789+0000 7f53637fe700 1 --2- 172.21.15.57:0/520011969 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f534c038850 0x7f534c03ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:29.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.790+0000 7f53637fe700 1 --2- 172.21.15.57:0/520011969 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f534c038850 0x7f534c03ad10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5358006fd0 tx=0x7f5358006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:29.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:29.794+0000 7f53617fa700 1 -- 172.21.15.57:0/520011969 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5354029360 con 0x7f5364108480 2023-07-20T18:52:30.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:30.006+0000 7f536a893700 1 -- 172.21.15.57:0/520011969 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}) v1 -- 0x7f53640008d0 con 0x7f534c038850 2023-07-20T18:52:30.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:30.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:30.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:30.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:30.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:30.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:52:30.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:30.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:52:30.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-20T18:52:30.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:30.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:30.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:31.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:30 smithi057 ceph-mon[126340]: Updating smithi088:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:52:31.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:30 smithi057 ceph-mon[126340]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:52:31.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:30 smithi057 ceph-mon[126340]: Updating smithi088:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T18:52:32.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:31 smithi057 ceph-mon[126340]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:52:32.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:31 smithi057 ceph-mon[126340]: Deploying cephadm binary to smithi192 2023-07-20T18:52:32.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:31 smithi057 ceph-mon[126340]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:32.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.361+0000 7f53617fa700 1 -- 172.21.15.57:0/520011969 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f53640008d0 con 0x7f534c038850 2023-07-20T18:52:32.401 INFO:teuthology.orchestra.run.smithi057.stdout:Added host 'smithi192' with addr '172.21.15.192' 2023-07-20T18:52:32.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.363+0000 7f536a893700 1 -- 172.21.15.57:0/520011969 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f534c038850 msgr2=0x7f534c03ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:32.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.363+0000 7f536a893700 1 --2- 172.21.15.57:0/520011969 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f534c038850 0x7f534c03ad10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5358006fd0 tx=0x7f5358006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:32.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.363+0000 7f536a893700 1 -- 172.21.15.57:0/520011969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5364108480 msgr2=0x7f5364074c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:32.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.363+0000 7f536a893700 1 --2- 172.21.15.57:0/520011969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5364108480 0x7f5364074c80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f535400b5c0 tx=0x7f5354005010 comp rx=0 tx=0).stop 2023-07-20T18:52:32.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.363+0000 7f536a893700 1 -- 172.21.15.57:0/520011969 shutdown_connections 2023-07-20T18:52:32.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.363+0000 7f536a893700 1 --2- 172.21.15.57:0/520011969 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f534c038850 0x7f534c03ad10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:32.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.363+0000 7f536a893700 1 --2- 172.21.15.57:0/520011969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5364108480 0x7f5364074c80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:32.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.363+0000 7f536a893700 1 -- 172.21.15.57:0/520011969 >> 172.21.15.57:0/520011969 conn(0x7f53640fdea0 msgr2=0x7f536410a810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:32.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.364+0000 7f536a893700 1 -- 172.21.15.57:0/520011969 shutdown_connections 2023-07-20T18:52:32.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:32.364+0000 7f536a893700 1 -- 172.21.15.57:0/520011969 wait complete. 2023-07-20T18:52:32.903 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T18:52:33.631 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:52:33.724 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:33.724 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:33 smithi057 ceph-mon[126340]: Added host smithi192 2023-07-20T18:52:33.725 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:33.725 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:33 smithi057 ceph-mon[126340]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:33.725 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:34.599 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:34 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:34.599 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:34 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:35.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.188+0000 7f3f1b700700 1 -- 172.21.15.57:0/943131479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f14100460 msgr2=0x7f3f14100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:35.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.188+0000 7f3f1b700700 1 --2- 172.21.15.57:0/943131479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f14100460 0x7f3f14100840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009910 tx=0x7f3f10009c20 comp rx=0 tx=0).stop 2023-07-20T18:52:35.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.188+0000 7f3f1b700700 1 -- 172.21.15.57:0/943131479 shutdown_connections 2023-07-20T18:52:35.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.188+0000 7f3f1b700700 1 --2- 172.21.15.57:0/943131479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f14100460 0x7f3f14100840 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:35.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.188+0000 7f3f1b700700 1 -- 172.21.15.57:0/943131479 >> 172.21.15.57:0/943131479 conn(0x7f3f140fbd30 msgr2=0x7f3f140fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:35.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.188+0000 7f3f1b700700 1 -- 172.21.15.57:0/943131479 shutdown_connections 2023-07-20T18:52:35.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.188+0000 7f3f1b700700 1 -- 172.21.15.57:0/943131479 wait complete. 2023-07-20T18:52:35.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.189+0000 7f3f1b700700 1 Processor -- start 2023-07-20T18:52:35.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.189+0000 7f3f1b700700 1 -- start start 2023-07-20T18:52:35.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.190+0000 7f3f1b700700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f14100460 0x7f3f1419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:35.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.190+0000 7f3f1b700700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1419dd00 con 0x7f3f14100460 2023-07-20T18:52:35.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.190+0000 7f3f1949c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f14100460 0x7f3f1419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:35.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.190+0000 7f3f1949c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f14100460 0x7f3f1419d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60180/0 (socket says 172.21.15.57:60180) 2023-07-20T18:52:35.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.190+0000 7f3f1949c700 1 -- 172.21.15.57:0/3330452010 learned_addr learned my addr 172.21.15.57:0/3330452010 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:35.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.190+0000 7f3f1949c700 1 -- 172.21.15.57:0/3330452010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f100095f0 con 0x7f3f14100460 2023-07-20T18:52:35.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.190+0000 7f3f1949c700 1 --2- 172.21.15.57:0/3330452010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f14100460 0x7f3f1419d7c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3f10005850 tx=0x7f3f10004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:35.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.191+0000 7f3f0a7fc700 1 -- 172.21.15.57:0/3330452010 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f1001d070 con 0x7f3f14100460 2023-07-20T18:52:35.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.191+0000 7f3f0a7fc700 1 -- 172.21.15.57:0/3330452010 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f3f10004500 con 0x7f3f14100460 2023-07-20T18:52:35.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.191+0000 7f3f0a7fc700 1 -- 172.21.15.57:0/3330452010 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f100039f0 con 0x7f3f14100460 2023-07-20T18:52:35.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.191+0000 7f3f1b700700 1 -- 172.21.15.57:0/3330452010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1419df00 con 0x7f3f14100460 2023-07-20T18:52:35.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.191+0000 7f3f1b700700 1 -- 172.21.15.57:0/3330452010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1419e2c0 con 0x7f3f14100460 2023-07-20T18:52:35.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.192+0000 7f3f1b700700 1 -- 172.21.15.57:0/3330452010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f14061da0 con 0x7f3f14100460 2023-07-20T18:52:35.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.192+0000 7f3f0a7fc700 1 -- 172.21.15.57:0/3330452010 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f3f1000b960 con 0x7f3f14100460 2023-07-20T18:52:35.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.193+0000 7f3f0a7fc700 1 --2- 172.21.15.57:0/3330452010 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3f00038800 0x7f3f0003acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:35.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.193+0000 7f3f0a7fc700 1 -- 172.21.15.57:0/3330452010 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3f1004ca00 con 0x7f3f14100460 2023-07-20T18:52:35.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.193+0000 7f3f18c9b700 1 --2- 172.21.15.57:0/3330452010 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3f00038800 0x7f3f0003acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:35.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.193+0000 7f3f18c9b700 1 --2- 172.21.15.57:0/3330452010 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3f00038800 0x7f3f0003acc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3f04006fd0 tx=0x7f3f04006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:35.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.198+0000 7f3f0a7fc700 1 -- 172.21.15.57:0/3330452010 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3f1002a540 con 0x7f3f14100460 2023-07-20T18:52:35.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.411+0000 7f3f1b700700 1 -- 172.21.15.57:0/3330452010 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3f14103de0 con 0x7f3f00038800 2023-07-20T18:52:35.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.412+0000 7f3f0a7fc700 1 -- 172.21.15.57:0/3330452010 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7f3f14103de0 con 0x7f3f00038800 2023-07-20T18:52:35.451 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:52:35.451 INFO:teuthology.orchestra.run.smithi057.stdout:[{"addr": "172.21.15.57", "hostname": "smithi057", "labels": [], "status": ""}, {"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2023-07-20T18:52:35.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.415+0000 7f3f1b700700 1 -- 172.21.15.57:0/3330452010 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3f00038800 msgr2=0x7f3f0003acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:35.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.415+0000 7f3f1b700700 1 --2- 172.21.15.57:0/3330452010 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3f00038800 0x7f3f0003acc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3f04006fd0 tx=0x7f3f04006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:35.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.415+0000 7f3f1b700700 1 -- 172.21.15.57:0/3330452010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f14100460 msgr2=0x7f3f1419d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:35.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.415+0000 7f3f1b700700 1 --2- 172.21.15.57:0/3330452010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f14100460 0x7f3f1419d7c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3f10005850 tx=0x7f3f10004ff0 comp rx=0 tx=0).stop 2023-07-20T18:52:35.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.415+0000 7f3f1b700700 1 -- 172.21.15.57:0/3330452010 shutdown_connections 2023-07-20T18:52:35.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.415+0000 7f3f1b700700 1 --2- 172.21.15.57:0/3330452010 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3f00038800 0x7f3f0003acc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:35.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.415+0000 7f3f1b700700 1 --2- 172.21.15.57:0/3330452010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f14100460 0x7f3f1419d7c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:35.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.415+0000 7f3f1b700700 1 -- 172.21.15.57:0/3330452010 >> 172.21.15.57:0/3330452010 conn(0x7f3f140fbd30 msgr2=0x7f3f141026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:35.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.415+0000 7f3f1b700700 1 -- 172.21.15.57:0/3330452010 shutdown_connections 2023-07-20T18:52:35.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:35.415+0000 7f3f1b700700 1 -- 172.21.15.57:0/3330452010 wait complete. 2023-07-20T18:52:35.892 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:35 smithi057 ceph-mon[126340]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:35.913 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-20T18:52:35.913 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph osd crush tunables default 2023-07-20T18:52:36.642 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:52:36.923 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:36 smithi057 ceph-mon[126340]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T18:52:38.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.138+0000 7f673c02e700 1 -- 172.21.15.57:0/403892561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6734072ba0 msgr2=0x7f6734072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:38.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.138+0000 7f673c02e700 1 --2- 172.21.15.57:0/403892561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6734072ba0 0x7f6734072f80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6730009910 tx=0x7f6730009c20 comp rx=0 tx=0).stop 2023-07-20T18:52:38.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.138+0000 7f673c02e700 1 -- 172.21.15.57:0/403892561 shutdown_connections 2023-07-20T18:52:38.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.138+0000 7f673c02e700 1 --2- 172.21.15.57:0/403892561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6734072ba0 0x7f6734072f80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:38.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.138+0000 7f673c02e700 1 -- 172.21.15.57:0/403892561 >> 172.21.15.57:0/403892561 conn(0x7f67340fbd30 msgr2=0x7f67340fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:38.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.139+0000 7f673c02e700 1 -- 172.21.15.57:0/403892561 shutdown_connections 2023-07-20T18:52:38.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.139+0000 7f673c02e700 1 -- 172.21.15.57:0/403892561 wait complete. 2023-07-20T18:52:38.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.139+0000 7f673c02e700 1 Processor -- start 2023-07-20T18:52:38.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.140+0000 7f673c02e700 1 -- start start 2023-07-20T18:52:38.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.140+0000 7f673c02e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6734072ba0 0x7f673419d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:38.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.140+0000 7f673c02e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673419dde0 con 0x7f6734072ba0 2023-07-20T18:52:38.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.140+0000 7f6739dca700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6734072ba0 0x7f673419d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:38.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.140+0000 7f6739dca700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6734072ba0 0x7f673419d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60196/0 (socket says 172.21.15.57:60196) 2023-07-20T18:52:38.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.140+0000 7f6739dca700 1 -- 172.21.15.57:0/1902958743 learned_addr learned my addr 172.21.15.57:0/1902958743 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:52:38.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.141+0000 7f6739dca700 1 -- 172.21.15.57:0/1902958743 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67300095f0 con 0x7f6734072ba0 2023-07-20T18:52:38.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.141+0000 7f6739dca700 1 --2- 172.21.15.57:0/1902958743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6734072ba0 0x7f673419d8a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6730005b40 tx=0x7f6730005010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:38.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.141+0000 7f672affd700 1 -- 172.21.15.57:0/1902958743 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f673001d070 con 0x7f6734072ba0 2023-07-20T18:52:38.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.141+0000 7f673c02e700 1 -- 172.21.15.57:0/1902958743 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673419dfe0 con 0x7f6734072ba0 2023-07-20T18:52:38.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.141+0000 7f672affd700 1 -- 172.21.15.57:0/1902958743 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f6730004500 con 0x7f6734072ba0 2023-07-20T18:52:38.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.141+0000 7f672affd700 1 -- 172.21.15.57:0/1902958743 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6730003ac0 con 0x7f6734072ba0 2023-07-20T18:52:38.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.141+0000 7f673c02e700 1 -- 172.21.15.57:0/1902958743 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673419e3a0 con 0x7f6734072ba0 2023-07-20T18:52:38.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.142+0000 7f672affd700 1 -- 172.21.15.57:0/1902958743 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f673000b960 con 0x7f6734072ba0 2023-07-20T18:52:38.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.142+0000 7f673c02e700 1 -- 172.21.15.57:0/1902958743 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6734061da0 con 0x7f6734072ba0 2023-07-20T18:52:38.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.143+0000 7f672affd700 1 --2- 172.21.15.57:0/1902958743 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6720038800 0x7f672003acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:38.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.143+0000 7f672affd700 1 -- 172.21.15.57:0/1902958743 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f673004c760 con 0x7f6734072ba0 2023-07-20T18:52:38.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.143+0000 7f67395c9700 1 --2- 172.21.15.57:0/1902958743 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6720038800 0x7f672003acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:38.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.144+0000 7f67395c9700 1 --2- 172.21.15.57:0/1902958743 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6720038800 0x7f672003acc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6724006fd0 tx=0x7f6724006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:38.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.148+0000 7f672affd700 1 -- 172.21.15.57:0/1902958743 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f673002a540 con 0x7f6734072ba0 2023-07-20T18:52:38.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:38.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:38.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:38.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:38.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:38.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:52:38.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:38.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:52:38.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: Updating smithi192:/etc/ceph/ceph.conf 2023-07-20T18:52:38.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:38.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:38.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:38.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.358+0000 7f673c02e700 1 -- 172.21.15.57:0/1902958743 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6734109760 con 0x7f6734072ba0 2023-07-20T18:52:38.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.804+0000 7f672affd700 1 -- 172.21.15.57:0/1902958743 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f673002a360 con 0x7f6734072ba0 2023-07-20T18:52:38.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.807+0000 7f673c02e700 1 -- 172.21.15.57:0/1902958743 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6720038800 msgr2=0x7f672003acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:38.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.807+0000 7f673c02e700 1 --2- 172.21.15.57:0/1902958743 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6720038800 0x7f672003acc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6724006fd0 tx=0x7f6724006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:38.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.807+0000 7f673c02e700 1 -- 172.21.15.57:0/1902958743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6734072ba0 msgr2=0x7f673419d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:38.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.807+0000 7f673c02e700 1 --2- 172.21.15.57:0/1902958743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6734072ba0 0x7f673419d8a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6730005b40 tx=0x7f6730005010 comp rx=0 tx=0).stop 2023-07-20T18:52:38.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.807+0000 7f673c02e700 1 -- 172.21.15.57:0/1902958743 shutdown_connections 2023-07-20T18:52:38.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.807+0000 7f673c02e700 1 --2- 172.21.15.57:0/1902958743 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6720038800 0x7f672003acc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:38.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.807+0000 7f673c02e700 1 --2- 172.21.15.57:0/1902958743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6734072ba0 0x7f673419d8a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:38.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.807+0000 7f673c02e700 1 -- 172.21.15.57:0/1902958743 >> 172.21.15.57:0/1902958743 conn(0x7f67340fbd30 msgr2=0x7f6734106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:38.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.808+0000 7f673c02e700 1 -- 172.21.15.57:0/1902958743 shutdown_connections 2023-07-20T18:52:38.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:52:38.808+0000 7f673c02e700 1 -- 172.21.15.57:0/1902958743 wait complete. 2023-07-20T18:52:38.848 INFO:teuthology.orchestra.run.smithi057.stderr:adjusted tunables profile to default 2023-07-20T18:52:39.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:38 smithi057 ceph-mon[126340]: Updating smithi192:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:52:39.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:38 smithi057 ceph-mon[126340]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:52:39.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:38 smithi057 ceph-mon[126340]: Updating smithi192:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T18:52:39.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:38 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1902958743' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T18:52:39.321 INFO:tasks.cephadm:Adding mon.a on smithi057 2023-07-20T18:52:39.322 INFO:tasks.cephadm:Adding mon.b on smithi088 2023-07-20T18:52:39.322 INFO:tasks.cephadm:Adding mon.c on smithi192 2023-07-20T18:52:39.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch apply mon '3;smithi057:172.21.15.57=a;smithi088:172.21.15.88=b;smithi192:172.21.15.192=c' 2023-07-20T18:52:39.905 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:52:40.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:39 smithi057 ceph-mon[126340]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:40.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:39 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1902958743' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T18:52:40.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:39 smithi057 ceph-mon[126340]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:52:41.250 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.210+0000 7f2af5617700 1 -- 172.21.15.192:0/3259219536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2af0104890 msgr2=0x7f2af0104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:41.250 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.210+0000 7f2af5617700 1 --2- 172.21.15.192:0/3259219536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2af0104890 0x7f2af0104c70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009910 tx=0x7f2ae0009c20 comp rx=0 tx=0).stop 2023-07-20T18:52:41.251 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.210+0000 7f2af5617700 1 -- 172.21.15.192:0/3259219536 shutdown_connections 2023-07-20T18:52:41.251 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.210+0000 7f2af5617700 1 --2- 172.21.15.192:0/3259219536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2af0104890 0x7f2af0104c70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:41.251 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.210+0000 7f2af5617700 1 -- 172.21.15.192:0/3259219536 >> 172.21.15.192:0/3259219536 conn(0x7f2af0075860 msgr2=0x7f2af0075c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:41.251 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.210+0000 7f2af5617700 1 -- 172.21.15.192:0/3259219536 shutdown_connections 2023-07-20T18:52:41.251 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.210+0000 7f2af5617700 1 -- 172.21.15.192:0/3259219536 wait complete. 2023-07-20T18:52:41.251 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.211+0000 7f2af5617700 1 Processor -- start 2023-07-20T18:52:41.251 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.211+0000 7f2af5617700 1 -- start start 2023-07-20T18:52:41.251 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.211+0000 7f2af5617700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2af0104890 0x7f2af01965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:41.252 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.211+0000 7f2af5617700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af0196ae0 con 0x7f2af0104890 2023-07-20T18:52:41.252 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.211+0000 7f2aeeffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2af0104890 0x7f2af01965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:41.252 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.211+0000 7f2aeeffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2af0104890 0x7f2af01965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.192:38030/0 (socket says 172.21.15.192:38030) 2023-07-20T18:52:41.252 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.211+0000 7f2aeeffd700 1 -- 172.21.15.192:0/3251164716 learned_addr learned my addr 172.21.15.192:0/3251164716 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-07-20T18:52:41.253 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.212+0000 7f2aeeffd700 1 -- 172.21.15.192:0/3251164716 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae00095f0 con 0x7f2af0104890 2023-07-20T18:52:41.253 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.212+0000 7f2aeeffd700 1 --2- 172.21.15.192:0/3251164716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2af0104890 0x7f2af01965a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0005850 tx=0x7f2ae0004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:41.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.212+0000 7f2ad7fff700 1 -- 172.21.15.192:0/3251164716 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ae001d070 con 0x7f2af0104890 2023-07-20T18:52:41.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.212+0000 7f2af5617700 1 -- 172.21.15.192:0/3251164716 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af0196c80 con 0x7f2af0104890 2023-07-20T18:52:41.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.212+0000 7f2ad7fff700 1 -- 172.21.15.192:0/3251164716 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f2ae0004500 con 0x7f2af0104890 2023-07-20T18:52:41.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.212+0000 7f2ad7fff700 1 -- 172.21.15.192:0/3251164716 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ae00039f0 con 0x7f2af0104890 2023-07-20T18:52:41.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.212+0000 7f2af5617700 1 -- 172.21.15.192:0/3251164716 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af0197100 con 0x7f2af0104890 2023-07-20T18:52:41.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.213+0000 7f2af5617700 1 -- 172.21.15.192:0/3251164716 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af0102b40 con 0x7f2af0104890 2023-07-20T18:52:41.254 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.213+0000 7f2ad7fff700 1 -- 172.21.15.192:0/3251164716 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f2ae000b960 con 0x7f2af0104890 2023-07-20T18:52:41.255 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.213+0000 7f2ad7fff700 1 --2- 172.21.15.192:0/3251164716 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ad80387b0 0x7f2ad803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:41.255 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.213+0000 7f2ad7fff700 1 -- 172.21.15.192:0/3251164716 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2ae004c790 con 0x7f2af0104890 2023-07-20T18:52:41.255 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.214+0000 7f2aee7fc700 1 --2- 172.21.15.192:0/3251164716 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ad80387b0 0x7f2ad803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:41.255 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.214+0000 7f2aee7fc700 1 --2- 172.21.15.192:0/3251164716 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ad80387b0 0x7f2ad803ac70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4006fd0 tx=0x7f2ae4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:41.257 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.219+0000 7f2ad7fff700 1 -- 172.21.15.192:0/3251164716 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2ae002a540 con 0x7f2af0104890 2023-07-20T18:52:41.451 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.412+0000 7f2af5617700 1 -- 172.21.15.192:0/3251164716 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi057:172.21.15.57=a;smithi088:172.21.15.88=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f2af0107e80 con 0x7f2ad80387b0 2023-07-20T18:52:41.464 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.425+0000 7f2ad7fff700 1 -- 172.21.15.192:0/3251164716 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2af0107e80 con 0x7f2ad80387b0 2023-07-20T18:52:41.464 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2023-07-20T18:52:41.466 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.427+0000 7f2af5617700 1 -- 172.21.15.192:0/3251164716 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ad80387b0 msgr2=0x7f2ad803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:41.466 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.427+0000 7f2af5617700 1 --2- 172.21.15.192:0/3251164716 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ad80387b0 0x7f2ad803ac70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4006fd0 tx=0x7f2ae4006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:41.466 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.427+0000 7f2af5617700 1 -- 172.21.15.192:0/3251164716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2af0104890 msgr2=0x7f2af01965a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:41.466 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.427+0000 7f2af5617700 1 --2- 172.21.15.192:0/3251164716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2af0104890 0x7f2af01965a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0005850 tx=0x7f2ae0004ff0 comp rx=0 tx=0).stop 2023-07-20T18:52:41.467 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.427+0000 7f2af5617700 1 -- 172.21.15.192:0/3251164716 shutdown_connections 2023-07-20T18:52:41.467 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.427+0000 7f2af5617700 1 --2- 172.21.15.192:0/3251164716 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ad80387b0 0x7f2ad803ac70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:41.467 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.427+0000 7f2af5617700 1 --2- 172.21.15.192:0/3251164716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2af0104890 0x7f2af01965a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:41.467 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.427+0000 7f2af5617700 1 -- 172.21.15.192:0/3251164716 >> 172.21.15.192:0/3251164716 conn(0x7f2af0075860 msgr2=0x7f2af00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:41.467 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.427+0000 7f2af5617700 1 -- 172.21.15.192:0/3251164716 shutdown_connections 2023-07-20T18:52:41.467 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:41.427+0000 7f2af5617700 1 -- 172.21.15.192:0/3251164716 wait complete. 2023-07-20T18:52:41.872 DEBUG:teuthology.orchestra.run.smithi088:mon.b> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.b.service 2023-07-20T18:52:41.875 DEBUG:teuthology.orchestra.run.smithi192:mon.c> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.c.service 2023-07-20T18:52:41.878 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T18:52:41.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T18:52:41.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:-- Logs begin at Thu 2023-07-20 18:24:33 UTC. -- 2023-07-20T18:52:41.910 INFO:journalctl@ceph.mon.b.smithi088.stdout:-- Logs begin at Thu 2023-07-20 18:25:07 UTC. -- 2023-07-20T18:52:42.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:41 smithi057 ceph-mon[126340]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:42.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:42.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:42.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:42.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:52:42.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:42.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:52:42.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:42.511 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:52:43.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:42 smithi057 ceph-mon[126340]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi057:172.21.15.57=a;smithi088:172.21.15.88=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:52:43.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:42 smithi057 ceph-mon[126340]: Saving service mon spec with placement smithi057:172.21.15.57=a;smithi088:172.21.15.88=b;smithi192:172.21.15.192=c;count:3 2023-07-20T18:52:43.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:42 smithi057 ceph-mon[126340]: Deploying daemon mon.c on smithi192 2023-07-20T18:52:43.919 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.879+0000 7fd103c2e700 1 -- 172.21.15.192:0/428300969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0fc0ff340 msgr2=0x7fd0fc0ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:43.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.879+0000 7fd103c2e700 1 --2- 172.21.15.192:0/428300969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0fc0ff340 0x7fd0fc0ff720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400b0a0 tx=0x7fd0f400b3b0 comp rx=0 tx=0).stop 2023-07-20T18:52:43.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.879+0000 7fd103c2e700 1 -- 172.21.15.192:0/428300969 shutdown_connections 2023-07-20T18:52:43.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.879+0000 7fd103c2e700 1 --2- 172.21.15.192:0/428300969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0fc0ff340 0x7fd0fc0ff720 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:43.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.879+0000 7fd103c2e700 1 -- 172.21.15.192:0/428300969 >> 172.21.15.192:0/428300969 conn(0x7fd0fc0fabf0 msgr2=0x7fd0fc0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:43.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.880+0000 7fd103c2e700 1 -- 172.21.15.192:0/428300969 shutdown_connections 2023-07-20T18:52:43.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.880+0000 7fd103c2e700 1 -- 172.21.15.192:0/428300969 wait complete. 2023-07-20T18:52:43.920 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.880+0000 7fd103c2e700 1 Processor -- start 2023-07-20T18:52:43.921 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.880+0000 7fd103c2e700 1 -- start start 2023-07-20T18:52:43.921 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.881+0000 7fd103c2e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0fc0ff340 0x7fd0fc196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:43.921 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.881+0000 7fd103c2e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc196b70 con 0x7fd0fc0ff340 2023-07-20T18:52:43.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.881+0000 7fd1019ca700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0fc0ff340 0x7fd0fc196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:43.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.881+0000 7fd1019ca700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0fc0ff340 0x7fd0fc196630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.192:38036/0 (socket says 172.21.15.192:38036) 2023-07-20T18:52:43.922 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.881+0000 7fd1019ca700 1 -- 172.21.15.192:0/4031346586 learned_addr learned my addr 172.21.15.192:0/4031346586 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-07-20T18:52:43.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.881+0000 7fd1019ca700 1 -- 172.21.15.192:0/4031346586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f4009d00 con 0x7fd0fc0ff340 2023-07-20T18:52:43.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.882+0000 7fd1019ca700 1 --2- 172.21.15.192:0/4031346586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0fc0ff340 0x7fd0fc196630 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009850 tx=0x7fd0f400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:43.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.882+0000 7fd0f2ffd700 1 -- 172.21.15.192:0/4031346586 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0f400e040 con 0x7fd0fc0ff340 2023-07-20T18:52:43.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.882+0000 7fd103c2e700 1 -- 172.21.15.192:0/4031346586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc196d70 con 0x7fd0fc0ff340 2023-07-20T18:52:43.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.882+0000 7fd0f2ffd700 1 -- 172.21.15.192:0/4031346586 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fd0f4003d40 con 0x7fd0fc0ff340 2023-07-20T18:52:43.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.882+0000 7fd0f2ffd700 1 -- 172.21.15.192:0/4031346586 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0f401d6e0 con 0x7fd0fc0ff340 2023-07-20T18:52:43.923 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.882+0000 7fd103c2e700 1 -- 172.21.15.192:0/4031346586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc197190 con 0x7fd0fc0ff340 2023-07-20T18:52:43.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.883+0000 7fd103c2e700 1 -- 172.21.15.192:0/4031346586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e0005320 con 0x7fd0fc0ff340 2023-07-20T18:52:43.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.883+0000 7fd0f2ffd700 1 -- 172.21.15.192:0/4031346586 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fd0f4019070 con 0x7fd0fc0ff340 2023-07-20T18:52:43.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.883+0000 7fd0f2ffd700 1 --2- 172.21.15.192:0/4031346586 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd0e80387b0 0x7fd0e803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:43.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.883+0000 7fd0f2ffd700 1 -- 172.21.15.192:0/4031346586 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd0f404c920 con 0x7fd0fc0ff340 2023-07-20T18:52:43.924 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.883+0000 7fd1011c9700 1 --2- 172.21.15.192:0/4031346586 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd0e80387b0 0x7fd0e803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:43.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.884+0000 7fd1011c9700 1 --2- 172.21.15.192:0/4031346586 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd0e80387b0 0x7fd0e803ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec006fd0 tx=0x7fd0ec006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:43.927 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:43.889+0000 7fd0f2ffd700 1 -- 172.21.15.192:0/4031346586 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd0f401bc10 con 0x7fd0fc0ff340 2023-07-20T18:52:44.197 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.158+0000 7fd103c2e700 1 -- 172.21.15.192:0/4031346586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0e00059f0 con 0x7fd0fc0ff340 2023-07-20T18:52:44.198 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.159+0000 7fd0f2ffd700 1 -- 172.21.15.192:0/4031346586 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7fd0f4017030 con 0x7fd0fc0ff340 2023-07-20T18:52:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:52:44.199 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","modified":"2023-07-20T18:50:56.856612Z","created":"2023-07-20T18:50:56.856612Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:3300","nonce":0},{"type":"v1","addr":"172.21.15.57:6789","nonce":0}]},"addr":"172.21.15.57:6789/0","public_addr":"172.21.15.57:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-20T18:52:44.200 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.161+0000 7fd103c2e700 1 -- 172.21.15.192:0/4031346586 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd0e80387b0 msgr2=0x7fd0e803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:44.201 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.161+0000 7fd103c2e700 1 --2- 172.21.15.192:0/4031346586 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd0e80387b0 0x7fd0e803ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec006fd0 tx=0x7fd0ec006e40 comp rx=0 tx=0).stop 2023-07-20T18:52:44.201 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.161+0000 7fd103c2e700 1 -- 172.21.15.192:0/4031346586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0fc0ff340 msgr2=0x7fd0fc196630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:44.201 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.161+0000 7fd103c2e700 1 --2- 172.21.15.192:0/4031346586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0fc0ff340 0x7fd0fc196630 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009850 tx=0x7fd0f400bd40 comp rx=0 tx=0).stop 2023-07-20T18:52:44.201 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.161+0000 7fd103c2e700 1 -- 172.21.15.192:0/4031346586 shutdown_connections 2023-07-20T18:52:44.201 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.161+0000 7fd103c2e700 1 --2- 172.21.15.192:0/4031346586 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd0e80387b0 0x7fd0e803ac70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:44.202 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.161+0000 7fd103c2e700 1 --2- 172.21.15.192:0/4031346586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0fc0ff340 0x7fd0fc196630 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:44.202 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.161+0000 7fd103c2e700 1 -- 172.21.15.192:0/4031346586 >> 172.21.15.192:0/4031346586 conn(0x7fd0fc0fabf0 msgr2=0x7fd0fc1035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:44.202 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.161+0000 7fd103c2e700 1 -- 172.21.15.192:0/4031346586 shutdown_connections 2023-07-20T18:52:44.202 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:44.161+0000 7fd103c2e700 1 -- 172.21.15.192:0/4031346586 wait complete. 2023-07-20T18:52:44.202 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2023-07-20T18:52:44.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:43 smithi057 ceph-mon[126340]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.079 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:44 smithi192 systemd[1]: Starting Ceph mon.c for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:52:45.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:44 smithi057 ceph-mon[126340]: from='client.? 172.21.15.192:0/4031346586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T18:52:45.510 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 podman[129160]: 2023-07-20T18:52:45.510 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T18:52:45.511 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T18:52:45.511 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pidfile_write: ignore empty --pid-file 2023-07-20T18:52:45.511 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 bash[129160]: ed109da9400692c65ea5a33bc2704786a73f3e569d462903c8e91c054850850e 2023-07-20T18:52:45.511 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: load: jerasure load: lrc 2023-07-20T18:52:45.511 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: RocksDB version: 7.9.2 2023-07-20T18:52:45.511 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Git sha 0 2023-07-20T18:52:45.511 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T18:52:45.511 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: DB SUMMARY 2023-07-20T18:52:45.512 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: DB Session ID: TOYU5ANVJSEGYIQKN50U 2023-07-20T18:52:45.512 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: CURRENT file: CURRENT 2023-07-20T18:52:45.512 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T18:52:45.512 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T18:52:45.512 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-20T18:52:45.512 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-20T18:52:45.512 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.error_if_exists: 0 2023-07-20T18:52:45.513 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.create_if_missing: 0 2023-07-20T18:52:45.513 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.paranoid_checks: 1 2023-07-20T18:52:45.513 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T18:52:45.513 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T18:52:45.513 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T18:52:45.513 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.env: 0x55f6ad5c5c60 2023-07-20T18:52:45.513 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T18:52:45.513 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.info_log: 0x55f6af6792c0 2023-07-20T18:52:45.514 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T18:52:45.514 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.statistics: (nil) 2023-07-20T18:52:45.514 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.use_fsync: 0 2023-07-20T18:52:45.514 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_log_file_size: 0 2023-07-20T18:52:45.514 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T18:52:45.514 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T18:52:45.514 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T18:52:45.514 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T18:52:45.515 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.allow_fallocate: 1 2023-07-20T18:52:45.515 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T18:52:45.515 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T18:52:45.515 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.use_direct_reads: 0 2023-07-20T18:52:45.515 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T18:52:45.515 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T18:52:45.515 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.db_log_dir: 2023-07-20T18:52:45.516 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.wal_dir: 2023-07-20T18:52:45.516 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T18:52:45.516 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T18:52:45.516 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T18:52:45.516 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T18:52:45.516 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T18:52:45.516 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T18:52:45.517 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T18:52:45.517 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T18:52:45.517 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.write_buffer_manager: 0x55f6af59a3c0 2023-07-20T18:52:45.517 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T18:52:45.517 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T18:52:45.517 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T18:52:45.517 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T18:52:45.518 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T18:52:45.518 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T18:52:45.518 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T18:52:45.518 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T18:52:45.518 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.unordered_write: 0 2023-07-20T18:52:45.519 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T18:52:45.519 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T18:52:45.519 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T18:52:45.519 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T18:52:45.519 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.row_cache: None 2023-07-20T18:52:45.520 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.wal_filter: None 2023-07-20T18:52:45.520 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T18:52:45.520 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T18:52:45.520 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.two_write_queues: 0 2023-07-20T18:52:45.520 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T18:52:45.520 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.wal_compression: 0 2023-07-20T18:52:45.520 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.atomic_flush: 0 2023-07-20T18:52:45.520 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T18:52:45.521 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T18:52:45.521 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T18:52:45.521 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.log_readahead_size: 0 2023-07-20T18:52:45.521 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T18:52:45.521 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T18:52:45.521 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T18:52:45.521 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T18:52:45.523 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T18:52:45.523 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T18:52:45.523 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T18:52:45.523 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_background_jobs: 2 2023-07-20T18:52:45.523 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_background_compactions: -1 2023-07-20T18:52:45.523 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_subcompactions: 1 2023-07-20T18:52:45.524 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T18:52:45.524 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T18:52:45.524 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T18:52:45.524 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T18:52:45.524 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T18:52:45.524 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T18:52:45.524 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T18:52:45.524 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T18:52:45.525 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_open_files: -1 2023-07-20T18:52:45.525 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T18:52:45.525 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T18:52:45.525 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T18:52:45.525 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T18:52:45.525 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_background_flushes: -1 2023-07-20T18:52:45.525 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Compression algorithms supported: 2023-07-20T18:52:45.526 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T18:52:45.526 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: kZSTD supported: 0 2023-07-20T18:52:45.526 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: kXpressCompression supported: 0 2023-07-20T18:52:45.526 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T18:52:45.526 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: kZlibCompression supported: 1 2023-07-20T18:52:45.526 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: kSnappyCompression supported: 1 2023-07-20T18:52:45.526 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: kLZ4Compression supported: 1 2023-07-20T18:52:45.526 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: kBZip2Compression supported: 0 2023-07-20T18:52:45.527 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T18:52:45.527 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T18:52:45.527 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-20T18:52:45.527 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T18:52:45.527 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T18:52:45.527 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.merge_operator: 2023-07-20T18:52:45.527 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_filter: None 2023-07-20T18:52:45.527 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T18:52:45.528 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T18:52:45.528 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T18:52:45.528 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T18:52:45.528 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f6af6793c0) 2023-07-20T18:52:45.528 INFO:journalctl@ceph.mon.c.smithi192.stdout: cache_index_and_filter_blocks: 1 2023-07-20T18:52:45.528 INFO:journalctl@ceph.mon.c.smithi192.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T18:52:45.529 INFO:journalctl@ceph.mon.c.smithi192.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T18:52:45.529 INFO:journalctl@ceph.mon.c.smithi192.stdout: pin_top_level_index_and_filter: 1 2023-07-20T18:52:45.529 INFO:journalctl@ceph.mon.c.smithi192.stdout: index_type: 0 2023-07-20T18:52:45.529 INFO:journalctl@ceph.mon.c.smithi192.stdout: data_block_index_type: 0 2023-07-20T18:52:45.529 INFO:journalctl@ceph.mon.c.smithi192.stdout: index_shortening: 1 2023-07-20T18:52:45.530 INFO:journalctl@ceph.mon.c.smithi192.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T18:52:45.530 INFO:journalctl@ceph.mon.c.smithi192.stdout: checksum: 4 2023-07-20T18:52:45.530 INFO:journalctl@ceph.mon.c.smithi192.stdout: no_block_cache: 0 2023-07-20T18:52:45.530 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache: 0x55f6af639350 2023-07-20T18:52:45.530 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache_name: BinnedLRUCache 2023-07-20T18:52:45.530 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache_options: 2023-07-20T18:52:45.530 INFO:journalctl@ceph.mon.c.smithi192.stdout: capacity : 536870912 2023-07-20T18:52:45.531 INFO:journalctl@ceph.mon.c.smithi192.stdout: num_shard_bits : 4 2023-07-20T18:52:45.531 INFO:journalctl@ceph.mon.c.smithi192.stdout: strict_capacity_limit : 0 2023-07-20T18:52:45.531 INFO:journalctl@ceph.mon.c.smithi192.stdout: high_pri_pool_ratio: 0.000 2023-07-20T18:52:45.531 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache_compressed: (nil) 2023-07-20T18:52:45.531 INFO:journalctl@ceph.mon.c.smithi192.stdout: persistent_cache: (nil) 2023-07-20T18:52:45.531 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_size: 4096 2023-07-20T18:52:45.531 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_size_deviation: 10 2023-07-20T18:52:45.532 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_restart_interval: 16 2023-07-20T18:52:45.532 INFO:journalctl@ceph.mon.c.smithi192.stdout: index_block_restart_interval: 1 2023-07-20T18:52:45.532 INFO:journalctl@ceph.mon.c.smithi192.stdout: metadata_block_size: 4096 2023-07-20T18:52:45.532 INFO:journalctl@ceph.mon.c.smithi192.stdout: partition_filters: 0 2023-07-20T18:52:45.532 INFO:journalctl@ceph.mon.c.smithi192.stdout: use_delta_encoding: 1 2023-07-20T18:52:45.532 INFO:journalctl@ceph.mon.c.smithi192.stdout: filter_policy: bloomfilter 2023-07-20T18:52:45.532 INFO:journalctl@ceph.mon.c.smithi192.stdout: whole_key_filtering: 1 2023-07-20T18:52:45.532 INFO:journalctl@ceph.mon.c.smithi192.stdout: verify_compression: 0 2023-07-20T18:52:45.533 INFO:journalctl@ceph.mon.c.smithi192.stdout: read_amp_bytes_per_bit: 0 2023-07-20T18:52:45.533 INFO:journalctl@ceph.mon.c.smithi192.stdout: format_version: 5 2023-07-20T18:52:45.533 INFO:journalctl@ceph.mon.c.smithi192.stdout: enable_index_compression: 1 2023-07-20T18:52:45.533 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_align: 0 2023-07-20T18:52:45.533 INFO:journalctl@ceph.mon.c.smithi192.stdout: max_auto_readahead_size: 262144 2023-07-20T18:52:45.533 INFO:journalctl@ceph.mon.c.smithi192.stdout: prepopulate_block_cache: 0 2023-07-20T18:52:45.533 INFO:journalctl@ceph.mon.c.smithi192.stdout: initial_auto_readahead_size: 8192 2023-07-20T18:52:45.533 INFO:journalctl@ceph.mon.c.smithi192.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T18:52:45.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T18:52:45.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T18:52:45.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compression: NoCompression 2023-07-20T18:52:45.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T18:52:45.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T18:52:45.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T18:52:45.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.num_levels: 7 2023-07-20T18:52:45.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T18:52:45.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T18:52:45.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T18:52:45.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T18:52:45.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T18:52:45.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T18:52:45.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T18:52:45.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T18:52:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T18:52:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T18:52:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T18:52:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T18:52:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T18:52:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T18:52:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T18:52:45.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T18:52:45.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T18:52:45.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T18:52:45.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T18:52:45.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T18:52:45.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T18:52:45.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T18:52:45.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T18:52:45.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T18:52:45.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T18:52:45.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T18:52:45.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T18:52:45.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T18:52:45.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T18:52:45.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T18:52:45.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T18:52:45.539 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T18:52:45.539 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T18:52:45.539 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T18:52:45.539 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T18:52:45.539 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T18:52:45.540 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T18:52:45.540 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T18:52:45.540 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T18:52:45.540 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T18:52:45.540 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T18:52:45.540 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T18:52:45.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T18:52:45.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T18:52:45.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T18:52:45.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T18:52:45.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T18:52:45.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T18:52:45.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T18:52:45.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T18:52:45.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T18:52:45.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T18:52:45.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T18:52:45.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.table_properties_collectors: 2023-07-20T18:52:45.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.inplace_update_support: 0 2023-07-20T18:52:45.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T18:52:45.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T18:52:45.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T18:52:45.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T18:52:45.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.bloom_locality: 0 2023-07-20T18:52:45.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.max_successive_merges: 0 2023-07-20T18:52:45.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T18:52:45.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T18:52:45.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T18:52:45.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T18:52:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.ttl: 2592000 2023-07-20T18:52:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T18:52:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T18:52:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T18:52:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.enable_blob_files: false 2023-07-20T18:52:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.min_blob_size: 0 2023-07-20T18:52:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T18:52:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T18:52:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T18:52:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T18:52:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T18:52:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T18:52:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T18:52:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T18:52:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T18:52:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T18:52:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a5b7e696-70b6-4d46-8a74-649ff34f3fcc 2023-07-20T18:52:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879165380838, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T18:52:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T18:52:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879165381238, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689879165, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a5b7e696-70b6-4d46-8a74-649ff34f3fcc", "db_session_id": "TOYU5ANVJSEGYIQKN50U", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T18:52:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879165381308, "job": 1, "event": "recovery_finished"} 2023-07-20T18:52:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T18:52:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T18:52:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f6af6d2000 2023-07-20T18:52:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: DB pointer 0x55f6af6be000 2023-07-20T18:52:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T18:52:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T18:52:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** DB Stats ** 2023-07-20T18:52:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T18:52:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T18:52:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T18:52:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T18:52:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T18:52:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T18:52:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T18:52:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2023-07-20T18:52:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** Compaction Stats [default] ** 2023-07-20T18:52:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T18:52:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T18:52:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T18:52:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T18:52:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T18:52:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2023-07-20T18:52:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** Compaction Stats [default] ** 2023-07-20T18:52:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T18:52:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T18:52:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T18:52:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2023-07-20T18:52:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T18:52:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2023-07-20T18:52:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T18:52:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T18:52:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T18:52:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T18:52:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T18:52:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T18:52:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T18:52:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T18:52:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T18:52:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout: Block cache BinnedLRUCache@0x55f6af639350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-20T18:52:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T18:52:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2023-07-20T18:52:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T18:52:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-20T18:52:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: using public_addr v2:172.21.15.192:0/0 -> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-07-20T18:52:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: starting mon.c rank -1 at public addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] at bind addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:52:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(???) e0 preinit fsid 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:52:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).mds e1 new map 2023-07-20T18:52:45.554 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-20T18:52:45.554 INFO:journalctl@ceph.mon.c.smithi192.stdout: e1 2023-07-20T18:52:45.554 INFO:journalctl@ceph.mon.c.smithi192.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T18:52:45.554 INFO:journalctl@ceph.mon.c.smithi192.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T18:52:45.554 INFO:journalctl@ceph.mon.c.smithi192.stdout: legacy client fscid: -1 2023-07-20T18:52:45.554 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2023-07-20T18:52:45.555 INFO:journalctl@ceph.mon.c.smithi192.stdout: No filesystems configured 2023-07-20T18:52:45.555 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T18:52:45.555 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T18:52:45.555 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T18:52:45.555 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T18:52:45.555 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T18:52:45.555 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T18:52:45.556 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T18:52:45.556 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:52:45.556 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:52:45.556 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:52:45.556 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.556 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Added host smithi088 2023-07-20T18:52:45.556 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:45.556 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.557 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.557 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.557 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.557 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.557 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T18:52:45.557 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.557 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.558 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.558 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.558 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.558 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:52:45.558 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:45.558 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:52:45.558 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-20T18:52:45.558 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.559 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.559 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.559 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Updating smithi088:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:52:45.559 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:52:45.559 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Updating smithi088:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T18:52:45.559 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:52:45.559 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Deploying cephadm binary to smithi192 2023-07-20T18:52:45.559 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.560 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.560 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Added host smithi192 2023-07-20T18:52:45.560 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:45.560 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.560 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T18:52:45.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:52:45.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:45.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:52:45.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Updating smithi192:/etc/ceph/ceph.conf 2023-07-20T18:52:45.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Updating smithi192:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:52:45.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:52:45.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Updating smithi192:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T18:52:45.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1902958743' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T18:52:45.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1902958743' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T18:52:45.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:52:45.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:45.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:45.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:52:45.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:45.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:52:45.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:45.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi057:172.21.15.57=a;smithi088:172.21.15.88=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:52:45.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Saving service mon spec with placement smithi057:172.21.15.57=a;smithi088:172.21.15.88=b;smithi192:172.21.15.192=c;count:3 2023-07-20T18:52:45.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: Deploying daemon mon.c on smithi192 2023-07-20T18:52:45.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:45.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: from='client.? 172.21.15.192:0/4031346586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T18:52:45.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T18:52:45.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta expand map: {default=false} 2023-07-20T18:52:45.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta from 'false' to 'false' 2023-07-20T18:52:45.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta expanded map: {default=false} 2023-07-20T18:52:45.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta expand map: {default=info} 2023-07-20T18:52:45.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta from 'info' to 'info' 2023-07-20T18:52:45.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta expanded map: {default=info} 2023-07-20T18:52:45.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta expand map: {default=daemon} 2023-07-20T18:52:45.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T18:52:45.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T18:52:45.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta expand map: {default=debug} 2023-07-20T18:52:45.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T18:52:45.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: expand_channel_meta expanded map: {default=debug} 2023-07-20T18:52:45.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 ceph-mon[129200]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-20T18:52:45.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:45 smithi192 systemd[1]: Started Ceph mon.c for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:52:45.778 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T18:52:45.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T18:52:46.332 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:52:50.510 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.471+0000 7f85f9974700 1 -- 172.21.15.192:0/1978237477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85dc0057d0 msgr2=0x7f85dc007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:50.511 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.471+0000 7f85f9974700 1 --2- 172.21.15.192:0/1978237477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85dc0057d0 0x7f85dc007c60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85f4076160 tx=0x7f85e800feb0 comp rx=0 tx=0).stop 2023-07-20T18:52:50.511 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.471+0000 7f85f9974700 1 -- 172.21.15.192:0/1978237477 shutdown_connections 2023-07-20T18:52:50.511 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.471+0000 7f85f9974700 1 --2- 172.21.15.192:0/1978237477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85dc0057d0 0x7f85dc007c60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:50.511 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.471+0000 7f85f9974700 1 -- 172.21.15.192:0/1978237477 >> 172.21.15.192:0/1978237477 conn(0x7f85f4074d20 msgr2=0x7f85f4075130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:50.511 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.471+0000 7f85f9974700 1 -- 172.21.15.192:0/1978237477 shutdown_connections 2023-07-20T18:52:50.512 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.471+0000 7f85f9974700 1 -- 172.21.15.192:0/1978237477 wait complete. 2023-07-20T18:52:50.513 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.472+0000 7f85f9974700 1 Processor -- start 2023-07-20T18:52:50.513 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.472+0000 7f85f9974700 1 -- start start 2023-07-20T18:52:50.513 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.472+0000 7f85f9974700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85dc0057d0 0x7f85f4193700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:50.513 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.472+0000 7f85f9974700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f4193c40 0x7f85f4197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:50.514 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.472+0000 7f85f9974700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f41941d0 con 0x7f85f4193c40 2023-07-20T18:52:50.514 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.472+0000 7f85f9974700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4194310 con 0x7f85dc0057d0 2023-07-20T18:52:50.514 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f2ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85dc0057d0 0x7f85f4193700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:50.514 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f2ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85dc0057d0 0x7f85f4193700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:48056/0 (socket says 172.21.15.192:48056) 2023-07-20T18:52:50.514 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f2ffd700 1 -- 172.21.15.192:0/1354678674 learned_addr learned my addr 172.21.15.192:0/1354678674 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-07-20T18:52:50.514 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f27fc700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f4193c40 0x7f85f4197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:50.514 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f2ffd700 1 -- 172.21.15.192:0/1354678674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85dc0057d0 msgr2=0x7f85f4193700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-20T18:52:50.515 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f2ffd700 1 -- 172.21.15.192:0/1354678674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85dc0057d0 msgr2=0x7f85f4193700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T18:52:50.515 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f2ffd700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85dc0057d0 0x7f85f4193700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T18:52:50.515 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f2ffd700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85dc0057d0 0x7f85f4193700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T18:52:50.515 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f27fc700 1 -- 172.21.15.192:0/1354678674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85dc0057d0 msgr2=0x7f85f4193700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T18:52:50.515 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f27fc700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85dc0057d0 0x7f85f4193700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:50.515 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f27fc700 1 -- 172.21.15.192:0/1354678674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e800f900 con 0x7f85f4193c40 2023-07-20T18:52:50.515 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.473+0000 7f85f27fc700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f4193c40 0x7f85f4197000 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f85e400b4d0 tx=0x7f85e400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:50.515 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.474+0000 7f85f8972700 1 -- 172.21.15.192:0/1354678674 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f85e4011630 con 0x7f85f4193c40 2023-07-20T18:52:50.516 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.474+0000 7f85f8972700 1 -- 172.21.15.192:0/1354678674 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f85e4011c70 con 0x7f85f4193c40 2023-07-20T18:52:50.516 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.474+0000 7f85f9974700 1 -- 172.21.15.192:0/1354678674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f4197540 con 0x7f85f4193c40 2023-07-20T18:52:50.516 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.474+0000 7f85f8972700 1 -- 172.21.15.192:0/1354678674 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f85e4010d90 con 0x7f85f4193c40 2023-07-20T18:52:50.516 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.474+0000 7f85f9974700 1 -- 172.21.15.192:0/1354678674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f4197a90 con 0x7f85f4193c40 2023-07-20T18:52:50.517 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.475+0000 7f85f8972700 1 -- 172.21.15.192:0/1354678674 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f85e401a400 con 0x7f85f4193c40 2023-07-20T18:52:50.517 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.475+0000 7f85f9974700 1 -- 172.21.15.192:0/1354678674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f4061da0 con 0x7f85f4193c40 2023-07-20T18:52:50.517 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.475+0000 7f85f8972700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f85e00385e0 0x7f85e003aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:50.517 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.475+0000 7f85f8972700 1 -- 172.21.15.192:0/1354678674 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f85e404ae40 con 0x7f85f4193c40 2023-07-20T18:52:50.517 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.475+0000 7f85f2ffd700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f85e00385e0 0x7f85e003aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:50.517 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.476+0000 7f85f2ffd700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f85e00385e0 0x7f85e003aaa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f85e8001eb0 tx=0x7f85e8005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:50.520 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.481+0000 7f85f8972700 1 -- 172.21.15.192:0/1354678674 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f85e4017680 con 0x7f85f4193c40 2023-07-20T18:52:50.782 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.743+0000 7f85f9974700 1 -- 172.21.15.192:0/1354678674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85f4197dc0 con 0x7f85f4193c40 2023-07-20T18:52:50.783 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.744+0000 7f85f8972700 1 -- 172.21.15.192:0/1354678674 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f85e40172c0 con 0x7f85f4193c40 2023-07-20T18:52:50.783 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:52:50.783 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","modified":"2023-07-20T18:52:45.411991Z","created":"2023-07-20T18:50:56.856612Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:3300","nonce":0},{"type":"v1","addr":"172.21.15.57:6789","nonce":0}]},"addr":"172.21.15.57:6789/0","public_addr":"172.21.15.57:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T18:52:50.785 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 -- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f85e00385e0 msgr2=0x7f85e003aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:50.785 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f85e00385e0 0x7f85e003aaa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f85e8001eb0 tx=0x7f85e8005960 comp rx=0 tx=0).stop 2023-07-20T18:52:50.785 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 -- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f4193c40 msgr2=0x7f85f4197000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:50.785 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f4193c40 0x7f85f4197000 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f85e400b4d0 tx=0x7f85e400b890 comp rx=0 tx=0).stop 2023-07-20T18:52:50.785 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 -- 172.21.15.192:0/1354678674 shutdown_connections 2023-07-20T18:52:50.786 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f85e00385e0 0x7f85e003aaa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:50.786 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85dc0057d0 0x7f85f4193700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:50.786 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 --2- 172.21.15.192:0/1354678674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f4193c40 0x7f85f4197000 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:50.786 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 -- 172.21.15.192:0/1354678674 >> 172.21.15.192:0/1354678674 conn(0x7f85f4074d20 msgr2=0x7f85f4102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:50.786 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 -- 172.21.15.192:0/1354678674 shutdown_connections 2023-07-20T18:52:50.786 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:50.746+0000 7f85f9974700 1 -- 172.21.15.192:0/1354678674 wait complete. 2023-07-20T18:52:50.787 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2023-07-20T18:52:50.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:52:50.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: mon.a calling monitor election 2023-07-20T18:52:50.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:50.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: mon.c calling monitor election 2023-07-20T18:52:50.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:50.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T18:52:50.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: monmap e2: 2 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2023-07-20T18:52:50.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: fsmap 2023-07-20T18:52:50.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:52:50.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: mgrmap e13: x(active, since 47s) 2023-07-20T18:52:50.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: overall HEALTH_OK 2023-07-20T18:52:50.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:50.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:50.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:52:50.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: mon.a calling monitor election 2023-07-20T18:52:50.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:50.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: mon.c calling monitor election 2023-07-20T18:52:50.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:50.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:50.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T18:52:50.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: monmap e2: 2 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2023-07-20T18:52:50.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: fsmap 2023-07-20T18:52:50.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:52:50.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: mgrmap e13: x(active, since 47s) 2023-07-20T18:52:50.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: overall HEALTH_OK 2023-07-20T18:52:50.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:50.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:51.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:51.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:52:51.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:51.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:51 smithi192 ceph-mon[129200]: Deploying daemon mon.b on smithi088 2023-07-20T18:52:51.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:51 smithi192 ceph-mon[129200]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:51.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:51 smithi192 ceph-mon[129200]: from='client.? 172.21.15.192:0/1354678674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T18:52:51.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:51.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:51.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:52:51.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:51.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:51 smithi057 ceph-mon[126340]: Deploying daemon mon.b on smithi088 2023-07-20T18:52:51.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:51 smithi057 ceph-mon[126340]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:51.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:51 smithi057 ceph-mon[126340]: from='client.? 172.21.15.192:0/1354678674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T18:52:51.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:52.191 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T18:52:52.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T18:52:52.755 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:52:53.370 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 systemd[1]: Starting Ceph mon.b for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:52:53.623 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 podman[128783]: 2023-07-20T18:52:53.623 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 bash[128783]: cc3eaefb27afd7296ac4951b720aaec38a7a434f1ec7bd3f4f16195dbe17e163 2023-07-20T18:52:53.623 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T18:52:53.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T18:52:53.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pidfile_write: ignore empty --pid-file 2023-07-20T18:52:53.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: load: jerasure load: lrc 2023-07-20T18:52:53.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: RocksDB version: 7.9.2 2023-07-20T18:52:53.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Git sha 0 2023-07-20T18:52:53.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T18:52:53.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: DB SUMMARY 2023-07-20T18:52:53.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: DB Session ID: 66775CX9DXXR5EE7K1GS 2023-07-20T18:52:53.625 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: CURRENT file: CURRENT 2023-07-20T18:52:53.625 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T18:52:53.625 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T18:52:53.625 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-20T18:52:53.625 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-20T18:52:53.625 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.error_if_exists: 0 2023-07-20T18:52:53.625 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.create_if_missing: 0 2023-07-20T18:52:53.626 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.paranoid_checks: 1 2023-07-20T18:52:53.626 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T18:52:53.626 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T18:52:53.626 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T18:52:53.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.env: 0x56429fe38c60 2023-07-20T18:52:53.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T18:52:53.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.info_log: 0x5642a0db72c0 2023-07-20T18:52:53.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T18:52:53.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.statistics: (nil) 2023-07-20T18:52:53.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.use_fsync: 0 2023-07-20T18:52:53.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_log_file_size: 0 2023-07-20T18:52:53.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T18:52:53.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T18:52:53.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T18:52:53.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T18:52:53.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.allow_fallocate: 1 2023-07-20T18:52:53.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T18:52:53.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T18:52:53.629 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.use_direct_reads: 0 2023-07-20T18:52:53.629 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T18:52:53.629 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T18:52:53.630 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.db_log_dir: 2023-07-20T18:52:53.630 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.wal_dir: 2023-07-20T18:52:53.630 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T18:52:53.630 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T18:52:53.630 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T18:52:53.630 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T18:52:53.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T18:52:53.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T18:52:53.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T18:52:53.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T18:52:53.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.write_buffer_manager: 0x5642a0cd83c0 2023-07-20T18:52:53.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T18:52:53.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T18:52:53.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T18:52:53.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T18:52:53.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T18:52:53.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T18:52:53.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T18:52:53.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T18:52:53.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.unordered_write: 0 2023-07-20T18:52:53.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T18:52:53.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T18:52:53.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T18:52:53.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T18:52:53.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.row_cache: None 2023-07-20T18:52:53.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.wal_filter: None 2023-07-20T18:52:53.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T18:52:53.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T18:52:53.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.two_write_queues: 0 2023-07-20T18:52:53.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T18:52:53.637 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.wal_compression: 0 2023-07-20T18:52:53.637 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.atomic_flush: 0 2023-07-20T18:52:53.637 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T18:52:53.637 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T18:52:53.637 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T18:52:53.637 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.log_readahead_size: 0 2023-07-20T18:52:53.637 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T18:52:53.638 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T18:52:53.638 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T18:52:53.638 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T18:52:53.638 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T18:52:53.638 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T18:52:53.638 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T18:52:53.638 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_background_jobs: 2 2023-07-20T18:52:53.638 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_background_compactions: -1 2023-07-20T18:52:53.639 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_subcompactions: 1 2023-07-20T18:52:53.639 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T18:52:53.639 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T18:52:53.639 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T18:52:53.639 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T18:52:53.639 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T18:52:53.639 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T18:52:53.640 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T18:52:53.640 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T18:52:53.640 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_open_files: -1 2023-07-20T18:52:53.640 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T18:52:53.641 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T18:52:53.641 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T18:52:53.641 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T18:52:53.641 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_background_flushes: -1 2023-07-20T18:52:53.641 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Compression algorithms supported: 2023-07-20T18:52:53.641 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T18:52:53.642 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: kZSTD supported: 0 2023-07-20T18:52:53.642 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: kXpressCompression supported: 0 2023-07-20T18:52:53.642 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T18:52:53.642 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: kZlibCompression supported: 1 2023-07-20T18:52:53.642 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: kSnappyCompression supported: 1 2023-07-20T18:52:53.642 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: kLZ4Compression supported: 1 2023-07-20T18:52:53.642 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: kBZip2Compression supported: 0 2023-07-20T18:52:53.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T18:52:53.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T18:52:53.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-20T18:52:53.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T18:52:53.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T18:52:53.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.merge_operator: 2023-07-20T18:52:53.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_filter: None 2023-07-20T18:52:53.644 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T18:52:53.644 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T18:52:53.644 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T18:52:53.644 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T18:52:53.644 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5642a0db73c0) 2023-07-20T18:52:53.645 INFO:journalctl@ceph.mon.b.smithi088.stdout: cache_index_and_filter_blocks: 1 2023-07-20T18:52:53.645 INFO:journalctl@ceph.mon.b.smithi088.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T18:52:53.645 INFO:journalctl@ceph.mon.b.smithi088.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T18:52:53.645 INFO:journalctl@ceph.mon.b.smithi088.stdout: pin_top_level_index_and_filter: 1 2023-07-20T18:52:53.645 INFO:journalctl@ceph.mon.b.smithi088.stdout: index_type: 0 2023-07-20T18:52:53.645 INFO:journalctl@ceph.mon.b.smithi088.stdout: data_block_index_type: 0 2023-07-20T18:52:53.645 INFO:journalctl@ceph.mon.b.smithi088.stdout: index_shortening: 1 2023-07-20T18:52:53.646 INFO:journalctl@ceph.mon.b.smithi088.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T18:52:53.646 INFO:journalctl@ceph.mon.b.smithi088.stdout: checksum: 4 2023-07-20T18:52:53.646 INFO:journalctl@ceph.mon.b.smithi088.stdout: no_block_cache: 0 2023-07-20T18:52:53.646 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_cache: 0x5642a0d77350 2023-07-20T18:52:53.646 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_cache_name: BinnedLRUCache 2023-07-20T18:52:53.646 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_cache_options: 2023-07-20T18:52:53.647 INFO:journalctl@ceph.mon.b.smithi088.stdout: capacity : 536870912 2023-07-20T18:52:53.647 INFO:journalctl@ceph.mon.b.smithi088.stdout: num_shard_bits : 4 2023-07-20T18:52:53.647 INFO:journalctl@ceph.mon.b.smithi088.stdout: strict_capacity_limit : 0 2023-07-20T18:52:53.647 INFO:journalctl@ceph.mon.b.smithi088.stdout: high_pri_pool_ratio: 0.000 2023-07-20T18:52:53.647 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_cache_compressed: (nil) 2023-07-20T18:52:53.647 INFO:journalctl@ceph.mon.b.smithi088.stdout: persistent_cache: (nil) 2023-07-20T18:52:53.648 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_size: 4096 2023-07-20T18:52:53.648 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_size_deviation: 10 2023-07-20T18:52:53.648 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_restart_interval: 16 2023-07-20T18:52:53.648 INFO:journalctl@ceph.mon.b.smithi088.stdout: index_block_restart_interval: 1 2023-07-20T18:52:53.648 INFO:journalctl@ceph.mon.b.smithi088.stdout: metadata_block_size: 4096 2023-07-20T18:52:53.648 INFO:journalctl@ceph.mon.b.smithi088.stdout: partition_filters: 0 2023-07-20T18:52:53.648 INFO:journalctl@ceph.mon.b.smithi088.stdout: use_delta_encoding: 1 2023-07-20T18:52:53.649 INFO:journalctl@ceph.mon.b.smithi088.stdout: filter_policy: bloomfilter 2023-07-20T18:52:53.649 INFO:journalctl@ceph.mon.b.smithi088.stdout: whole_key_filtering: 1 2023-07-20T18:52:53.649 INFO:journalctl@ceph.mon.b.smithi088.stdout: verify_compression: 0 2023-07-20T18:52:53.649 INFO:journalctl@ceph.mon.b.smithi088.stdout: read_amp_bytes_per_bit: 0 2023-07-20T18:52:53.649 INFO:journalctl@ceph.mon.b.smithi088.stdout: format_version: 5 2023-07-20T18:52:53.649 INFO:journalctl@ceph.mon.b.smithi088.stdout: enable_index_compression: 1 2023-07-20T18:52:53.649 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_align: 0 2023-07-20T18:52:53.650 INFO:journalctl@ceph.mon.b.smithi088.stdout: max_auto_readahead_size: 262144 2023-07-20T18:52:53.650 INFO:journalctl@ceph.mon.b.smithi088.stdout: prepopulate_block_cache: 0 2023-07-20T18:52:53.650 INFO:journalctl@ceph.mon.b.smithi088.stdout: initial_auto_readahead_size: 8192 2023-07-20T18:52:53.650 INFO:journalctl@ceph.mon.b.smithi088.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T18:52:53.650 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T18:52:53.650 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T18:52:53.650 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compression: NoCompression 2023-07-20T18:52:53.651 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T18:52:53.651 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T18:52:53.651 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T18:52:53.651 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.num_levels: 7 2023-07-20T18:52:53.652 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T18:52:53.652 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T18:52:53.652 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T18:52:53.652 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T18:52:53.652 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T18:52:53.652 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T18:52:53.654 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T18:52:53.654 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T18:52:53.654 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T18:52:53.654 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T18:52:53.654 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T18:52:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T18:52:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T18:52:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T18:52:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T18:52:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T18:52:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T18:52:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T18:52:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T18:52:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T18:52:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T18:52:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T18:52:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T18:52:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T18:52:53.657 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T18:52:53.657 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T18:52:53.657 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T18:52:53.657 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T18:52:53.657 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T18:52:53.657 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T18:52:53.657 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T18:52:53.657 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T18:52:53.658 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T18:52:53.658 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T18:52:53.658 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T18:52:53.658 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T18:52:53.658 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T18:52:53.658 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T18:52:53.658 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T18:52:53.658 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T18:52:53.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T18:52:53.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T18:52:53.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T18:52:53.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T18:52:53.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T18:52:53.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T18:52:53.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T18:52:53.660 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T18:52:53.660 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T18:52:53.660 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T18:52:53.660 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T18:52:53.660 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T18:52:53.660 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T18:52:53.660 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.table_properties_collectors: 2023-07-20T18:52:53.660 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.inplace_update_support: 0 2023-07-20T18:52:53.661 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T18:52:53.661 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T18:52:53.661 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T18:52:53.661 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T18:52:53.661 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.bloom_locality: 0 2023-07-20T18:52:53.662 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.max_successive_merges: 0 2023-07-20T18:52:53.662 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T18:52:53.662 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T18:52:53.662 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T18:52:53.662 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T18:52:53.662 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.ttl: 2592000 2023-07-20T18:52:53.662 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T18:52:53.663 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T18:52:53.663 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T18:52:53.663 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.enable_blob_files: false 2023-07-20T18:52:53.663 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.min_blob_size: 0 2023-07-20T18:52:53.663 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T18:52:53.663 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T18:52:53.663 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T18:52:53.663 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T18:52:53.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T18:52:53.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T18:52:53.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T18:52:53.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T18:52:53.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T18:52:53.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T18:52:53.665 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ac288480-111b-43d8-9344-37f139521747 2023-07-20T18:52:53.665 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879173519784, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T18:52:53.665 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T18:52:53.665 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879173520198, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689879173, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ac288480-111b-43d8-9344-37f139521747", "db_session_id": "66775CX9DXXR5EE7K1GS", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T18:52:53.665 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879173520265, "job": 1, "event": "recovery_finished"} 2023-07-20T18:52:53.665 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T18:52:53.666 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T18:52:53.666 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5642a0e10000 2023-07-20T18:52:53.666 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: DB pointer 0x5642a0dfc000 2023-07-20T18:52:53.666 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T18:52:53.666 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T18:52:53.666 INFO:journalctl@ceph.mon.b.smithi088.stdout: ** DB Stats ** 2023-07-20T18:52:53.666 INFO:journalctl@ceph.mon.b.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T18:52:53.667 INFO:journalctl@ceph.mon.b.smithi088.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T18:52:53.667 INFO:journalctl@ceph.mon.b.smithi088.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T18:52:53.667 INFO:journalctl@ceph.mon.b.smithi088.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T18:52:53.667 INFO:journalctl@ceph.mon.b.smithi088.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T18:52:53.667 INFO:journalctl@ceph.mon.b.smithi088.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T18:52:53.667 INFO:journalctl@ceph.mon.b.smithi088.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T18:52:53.668 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-20T18:52:53.668 INFO:journalctl@ceph.mon.b.smithi088.stdout: ** Compaction Stats [default] ** 2023-07-20T18:52:53.668 INFO:journalctl@ceph.mon.b.smithi088.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T18:52:53.668 INFO:journalctl@ceph.mon.b.smithi088.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T18:52:53.668 INFO:journalctl@ceph.mon.b.smithi088.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T18:52:53.668 INFO:journalctl@ceph.mon.b.smithi088.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T18:52:53.668 INFO:journalctl@ceph.mon.b.smithi088.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T18:52:53.669 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-20T18:52:53.669 INFO:journalctl@ceph.mon.b.smithi088.stdout: ** Compaction Stats [default] ** 2023-07-20T18:52:53.669 INFO:journalctl@ceph.mon.b.smithi088.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T18:52:53.669 INFO:journalctl@ceph.mon.b.smithi088.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T18:52:53.669 INFO:journalctl@ceph.mon.b.smithi088.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T18:52:53.669 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-20T18:52:53.669 INFO:journalctl@ceph.mon.b.smithi088.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T18:52:53.670 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-20T18:52:53.670 INFO:journalctl@ceph.mon.b.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T18:52:53.670 INFO:journalctl@ceph.mon.b.smithi088.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T18:52:53.670 INFO:journalctl@ceph.mon.b.smithi088.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T18:52:53.670 INFO:journalctl@ceph.mon.b.smithi088.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T18:52:53.670 INFO:journalctl@ceph.mon.b.smithi088.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T18:52:53.670 INFO:journalctl@ceph.mon.b.smithi088.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T18:52:53.670 INFO:journalctl@ceph.mon.b.smithi088.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T18:52:53.671 INFO:journalctl@ceph.mon.b.smithi088.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T18:52:53.671 INFO:journalctl@ceph.mon.b.smithi088.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T18:52:53.671 INFO:journalctl@ceph.mon.b.smithi088.stdout: Block cache BinnedLRUCache@0x5642a0d77350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-20T18:52:53.671 INFO:journalctl@ceph.mon.b.smithi088.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T18:52:53.671 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-20T18:52:53.671 INFO:journalctl@ceph.mon.b.smithi088.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T18:52:53.672 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-20T18:52:53.672 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: using public_addr v2:172.21.15.88:0/0 -> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-07-20T18:52:53.672 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: starting mon.b rank -1 at public addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] at bind addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:52:53.672 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(???) e0 preinit fsid 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:52:53.673 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 systemd[1]: Started Ceph mon.b for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:52:53.673 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).mds e1 new map 2023-07-20T18:52:53.673 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-20T18:52:53.673 INFO:journalctl@ceph.mon.b.smithi088.stdout: e1 2023-07-20T18:52:53.673 INFO:journalctl@ceph.mon.b.smithi088.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T18:52:53.673 INFO:journalctl@ceph.mon.b.smithi088.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T18:52:53.673 INFO:journalctl@ceph.mon.b.smithi088.stdout: legacy client fscid: -1 2023-07-20T18:52:53.674 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-20T18:52:53.674 INFO:journalctl@ceph.mon.b.smithi088.stdout: No filesystems configured 2023-07-20T18:52:53.674 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T18:52:53.674 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T18:52:53.674 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T18:52:53.674 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T18:52:53.674 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T18:52:53.675 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T18:52:53.675 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T18:52:53.675 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:52:53.675 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:52:53.675 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:52:53.675 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.675 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Added host smithi088 2023-07-20T18:52:53.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:53.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T18:52:53.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:52:53.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:53.678 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:52:53.678 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-20T18:52:53.678 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.678 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.678 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.678 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Updating smithi088:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:52:53.678 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:52:53.679 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Updating smithi088:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T18:52:53.679 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:52:53.679 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Deploying cephadm binary to smithi192 2023-07-20T18:52:53.679 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.679 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.679 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Added host smithi192 2023-07-20T18:52:53.679 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:53.679 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.680 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.680 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.680 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.680 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.680 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T18:52:53.680 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.680 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.681 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.681 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.681 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.681 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:52:53.681 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:53.681 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:52:53.681 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Updating smithi192:/etc/ceph/ceph.conf 2023-07-20T18:52:53.682 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.682 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.682 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.682 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Updating smithi192:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:52:53.682 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:52:53.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Updating smithi192:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T18:52:53.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1902958743' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T18:52:53.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1902958743' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T18:52:53.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:52:53.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:53.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:53.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:52:53.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:52:53.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:53.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi057:172.21.15.57=a;smithi088:172.21.15.88=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:52:53.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Saving service mon spec with placement smithi057:172.21.15.57=a;smithi088:172.21.15.88=b;smithi192:172.21.15.192=c;count:3 2023-07-20T18:52:53.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Deploying daemon mon.c on smithi192 2023-07-20T18:52:53.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='client.? 172.21.15.192:0/4031346586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T18:52:53.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:52:53.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:53.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.a calling monitor election 2023-07-20T18:52:53.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:53.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:53.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.c calling monitor election 2023-07-20T18:52:53.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:53.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:53.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:53.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T18:52:53.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: monmap e2: 2 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2023-07-20T18:52:53.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: fsmap 2023-07-20T18:52:53.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:52:53.688 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mgrmap e13: x(active, since 47s) 2023-07-20T18:52:53.688 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: overall HEALTH_OK 2023-07-20T18:52:53.688 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.688 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.688 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:53.688 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:52:53.688 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:52:53.689 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: Deploying daemon mon.b on smithi088 2023-07-20T18:52:53.689 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:53.689 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='client.? 172.21.15.192:0/1354678674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T18:52:53.689 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:53.689 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T18:52:53.689 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-20T18:52:53.689 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta expand map: {default=false} 2023-07-20T18:52:53.689 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta from 'false' to 'false' 2023-07-20T18:52:53.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta expanded map: {default=false} 2023-07-20T18:52:53.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta expand map: {default=info} 2023-07-20T18:52:53.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta from 'info' to 'info' 2023-07-20T18:52:53.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta expanded map: {default=info} 2023-07-20T18:52:53.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta expand map: {default=daemon} 2023-07-20T18:52:53.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T18:52:53.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T18:52:53.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta expand map: {default=debug} 2023-07-20T18:52:53.691 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T18:52:53.691 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:52:53 smithi088 ceph-mon[128823]: expand_channel_meta expanded map: {default=debug} 2023-07-20T18:52:58.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:52:58.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: mon.a calling monitor election 2023-07-20T18:52:58.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: mon.c calling monitor election 2023-07-20T18:52:58.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:58.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:58.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:58.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:58.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:58.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:58.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:58.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:58.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:58.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T18:52:58.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: monmap e3: 3 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2023-07-20T18:52:58.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: fsmap 2023-07-20T18:52:58.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:52:58.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: mgrmap e13: x(active, since 55s) 2023-07-20T18:52:58.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T18:52:58.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T18:52:58.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T18:52:58.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: mon.b (rank 2) addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] is down (out of quorum) 2023-07-20T18:52:58.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:58.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:58.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:58.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:59.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:52:59.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: mon.a calling monitor election 2023-07-20T18:52:59.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: mon.c calling monitor election 2023-07-20T18:52:59.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:59.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:52:59.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:59.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:59.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:59.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:59.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:59.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:59.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:52:59.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T18:52:59.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: monmap e3: 3 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2023-07-20T18:52:59.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: fsmap 2023-07-20T18:52:59.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:52:59.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: mgrmap e13: x(active, since 55s) 2023-07-20T18:52:59.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T18:52:59.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T18:52:59.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T18:52:59.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: mon.b (rank 2) addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] is down (out of quorum) 2023-07-20T18:52:59.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:59.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:59.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:59.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:52:59.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.900+0000 7f3e959ea700 1 -- 172.21.15.192:0/4088240140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e780084b0 msgr2=0x7f3e7800a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:52:59.941 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.900+0000 7f3e959ea700 1 --2- 172.21.15.192:0/4088240140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e780084b0 0x7f3e7800a920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3e90075870 tx=0x7f3e8400cea0 comp rx=0 tx=0).stop 2023-07-20T18:52:59.941 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.900+0000 7f3e959ea700 1 -- 172.21.15.192:0/4088240140 shutdown_connections 2023-07-20T18:52:59.941 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.900+0000 7f3e959ea700 1 --2- 172.21.15.192:0/4088240140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e780084b0 0x7f3e7800a920 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:59.941 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.900+0000 7f3e959ea700 1 -- 172.21.15.192:0/4088240140 >> 172.21.15.192:0/4088240140 conn(0x7f3e900744c0 msgr2=0x7f3e900748d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:52:59.941 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.900+0000 7f3e959ea700 1 -- 172.21.15.192:0/4088240140 shutdown_connections 2023-07-20T18:52:59.941 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.900+0000 7f3e959ea700 1 -- 172.21.15.192:0/4088240140 wait complete. 2023-07-20T18:52:59.943 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.901+0000 7f3e959ea700 1 Processor -- start 2023-07-20T18:52:59.943 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.901+0000 7f3e959ea700 1 -- start start 2023-07-20T18:52:59.943 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.901+0000 7f3e959ea700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e780084b0 0x7f3e9019da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:59.944 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.901+0000 7f3e959ea700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9019dfa0 0x7f3e90197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:59.944 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.901+0000 7f3e959ea700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e90198020 0x7f3e901984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:59.944 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.901+0000 7f3e959ea700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9019e600 con 0x7f3e780084b0 2023-07-20T18:52:59.944 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.901+0000 7f3e959ea700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e901a4880 con 0x7f3e90198020 2023-07-20T18:52:59.944 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.901+0000 7f3e959ea700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e901a4a10 con 0x7f3e9019dfa0 2023-07-20T18:52:59.944 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8e7fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9019dfa0 0x7f3e90197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:59.944 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8e7fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9019dfa0 0x7f3e90197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:48086/0 (socket says 172.21.15.192:48086) 2023-07-20T18:52:59.945 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8e7fc700 1 -- 172.21.15.192:0/1945900421 learned_addr learned my addr 172.21.15.192:0/1945900421 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-07-20T18:52:59.945 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8e7fc700 1 -- 172.21.15.192:0/1945900421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9019dfa0 msgr2=0x7f3e90197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-20T18:52:59.945 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8e7fc700 1 -- 172.21.15.192:0/1945900421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9019dfa0 msgr2=0x7f3e90197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T18:52:59.945 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8e7fc700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9019dfa0 0x7f3e90197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T18:52:59.945 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8f7fe700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e90198020 0x7f3e901984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:59.945 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8effd700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e780084b0 0x7f3e9019da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:59.946 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8e7fc700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9019dfa0 0x7f3e90197ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T18:52:59.946 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8f7fe700 1 -- 172.21.15.192:0/1945900421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e90198020 msgr2=0x7f3e901984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-20T18:52:59.946 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8f7fe700 1 -- 172.21.15.192:0/1945900421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e90198020 msgr2=0x7f3e901984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T18:52:59.946 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8f7fe700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e90198020 0x7f3e901984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T18:52:59.946 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8f7fe700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e90198020 0x7f3e901984d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T18:52:59.947 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8effd700 1 -- 172.21.15.192:0/1945900421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9019dfa0 msgr2=0x7f3e90197ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T18:52:59.947 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8effd700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9019dfa0 0x7f3e90197ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:59.947 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8effd700 1 -- 172.21.15.192:0/1945900421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e90198020 msgr2=0x7f3e901984d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T18:52:59.947 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8effd700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e90198020 0x7f3e901984d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:52:59.947 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8effd700 1 -- 172.21.15.192:0/1945900421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8400c8f0 con 0x7f3e780084b0 2023-07-20T18:52:59.947 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.902+0000 7f3e8effd700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e780084b0 0x7f3e9019da60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3e900ffae0 tx=0x7f3e84019d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:59.947 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.903+0000 7f3e949e8700 1 -- 172.21.15.192:0/1945900421 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84005450 con 0x7f3e780084b0 2023-07-20T18:52:59.948 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.903+0000 7f3e959ea700 1 -- 172.21.15.192:0/1945900421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e901a4c90 con 0x7f3e780084b0 2023-07-20T18:52:59.949 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.903+0000 7f3e949e8700 1 -- 172.21.15.192:0/1945900421 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f3e84019810 con 0x7f3e780084b0 2023-07-20T18:52:59.949 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.903+0000 7f3e949e8700 1 -- 172.21.15.192:0/1945900421 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8401a960 con 0x7f3e780084b0 2023-07-20T18:52:59.949 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.903+0000 7f3e959ea700 1 -- 172.21.15.192:0/1945900421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e901a5180 con 0x7f3e780084b0 2023-07-20T18:52:59.949 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.904+0000 7f3e959ea700 1 -- 172.21.15.192:0/1945900421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9018dcf0 con 0x7f3e780084b0 2023-07-20T18:52:59.949 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.904+0000 7f3e949e8700 1 -- 172.21.15.192:0/1945900421 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f3e8402a070 con 0x7f3e780084b0 2023-07-20T18:52:59.949 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.904+0000 7f3e949e8700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e7c038c30 0x7f3e7c03b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:52:59.950 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.904+0000 7f3e949e8700 1 -- 172.21.15.192:0/1945900421 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3e8404e390 con 0x7f3e780084b0 2023-07-20T18:52:59.950 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.904+0000 7f3e8e7fc700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e7c038c30 0x7f3e7c03b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:52:59.950 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.905+0000 7f3e8e7fc700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e7c038c30 0x7f3e7c03b0f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009180 tx=0x7f3e80008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:52:59.950 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:52:59.910+0000 7f3e949e8700 1 -- 172.21.15.192:0/1945900421 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3e8400d3e0 con 0x7f3e780084b0 2023-07-20T18:52:59.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:59 smithi057 ceph-mon[126340]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:52:59.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:59 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:52:59.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:52:59 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:00.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:59 smithi192 ceph-mon[129200]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:00.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:59 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:00.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:52:59 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:00.224 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.183+0000 7f3e959ea700 1 -- 172.21.15.192:0/1945900421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e90061da0 con 0x7f3e780084b0 2023-07-20T18:53:00.225 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.184+0000 7f3e949e8700 1 -- 172.21.15.192:0/1945900421 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1270 (secure 0 0 0) 0x7f3e8400d3e0 con 0x7f3e780084b0 2023-07-20T18:53:00.225 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-20T18:53:00.225 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":3,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","modified":"2023-07-20T18:52:53.552108Z","created":"2023-07-20T18:50:56.856612Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:3300","nonce":0},{"type":"v1","addr":"172.21.15.57:6789","nonce":0}]},"addr":"172.21.15.57:6789/0","public_addr":"172.21.15.57:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T18:53:00.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 -- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e7c038c30 msgr2=0x7f3e7c03b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:00.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e7c038c30 0x7f3e7c03b0f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009180 tx=0x7f3e80008040 comp rx=0 tx=0).stop 2023-07-20T18:53:00.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 -- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e780084b0 msgr2=0x7f3e9019da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:00.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e780084b0 0x7f3e9019da60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3e900ffae0 tx=0x7f3e84019d50 comp rx=0 tx=0).stop 2023-07-20T18:53:00.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 -- 172.21.15.192:0/1945900421 shutdown_connections 2023-07-20T18:53:00.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e7c038c30 0x7f3e7c03b0f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:00.228 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e90198020 0x7f3e901984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:00.228 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e780084b0 0x7f3e9019da60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:00.228 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 --2- 172.21.15.192:0/1945900421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9019dfa0 0x7f3e90197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:00.228 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 -- 172.21.15.192:0/1945900421 >> 172.21.15.192:0/1945900421 conn(0x7f3e900744c0 msgr2=0x7f3e900fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:53:00.228 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 -- 172.21.15.192:0/1945900421 shutdown_connections 2023-07-20T18:53:00.228 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:00.186+0000 7f3e959ea700 1 -- 172.21.15.192:0/1945900421 wait complete. 2023-07-20T18:53:00.229 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 3 2023-07-20T18:53:00.629 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-20T18:53:00.629 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph config generate-minimal-conf 2023-07-20T18:53:00.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:53:00.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: mon.a calling monitor election 2023-07-20T18:53:00.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: mon.c calling monitor election 2023-07-20T18:53:00.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:00.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:53:00.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:00.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:00.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:00.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:00.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:00.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:00.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:00.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T18:53:00.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: monmap e3: 3 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2023-07-20T18:53:00.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: fsmap 2023-07-20T18:53:00.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:53:00.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: mgrmap e13: x(active, since 55s) 2023-07-20T18:53:00.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T18:53:00.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T18:53:00.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T18:53:00.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: mon.b (rank 2) addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] is down (out of quorum) 2023-07-20T18:53:00.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:00.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:00.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:00.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:00.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:00.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:00.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:01.387 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:53:01.693 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: mon.b calling monitor election 2023-07-20T18:53:01.694 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: Updating smithi057:/etc/ceph/ceph.conf 2023-07-20T18:53:01.694 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-20T18:53:01.694 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: Updating smithi192:/etc/ceph/ceph.conf 2023-07-20T18:53:01.694 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: Updating smithi088:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:53:01.694 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: Updating smithi192:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:53:01.695 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: mon.b calling monitor election 2023-07-20T18:53:01.695 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: mon.c calling monitor election 2023-07-20T18:53:01.695 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: mon.a calling monitor election 2023-07-20T18:53:01.696 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T18:53:01.696 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: monmap e3: 3 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2023-07-20T18:53:01.696 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: fsmap 2023-07-20T18:53:01.696 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:53:01.696 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: mgrmap e13: x(active, since 57s) 2023-07-20T18:53:01.697 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T18:53:01.697 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: Cluster is now healthy 2023-07-20T18:53:01.697 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: overall HEALTH_OK 2023-07-20T18:53:01.697 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:01.697 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.697 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.698 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.698 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.698 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: Updating smithi057:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:53:01.698 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.698 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.698 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: mon.b calling monitor election 2023-07-20T18:53:01.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: Updating smithi057:/etc/ceph/ceph.conf 2023-07-20T18:53:01.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-20T18:53:01.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: Updating smithi192:/etc/ceph/ceph.conf 2023-07-20T18:53:01.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: Updating smithi088:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:53:01.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: Updating smithi192:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:53:01.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: mon.b calling monitor election 2023-07-20T18:53:01.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: mon.c calling monitor election 2023-07-20T18:53:01.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: mon.a calling monitor election 2023-07-20T18:53:01.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T18:53:01.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: monmap e3: 3 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2023-07-20T18:53:01.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: fsmap 2023-07-20T18:53:01.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:53:01.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: mgrmap e13: x(active, since 57s) 2023-07-20T18:53:01.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T18:53:01.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: Cluster is now healthy 2023-07-20T18:53:01.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: overall HEALTH_OK 2023-07-20T18:53:01.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:01.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: Updating smithi057:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:53:01.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: mon.b calling monitor election 2023-07-20T18:53:01.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: Updating smithi057:/etc/ceph/ceph.conf 2023-07-20T18:53:01.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-20T18:53:01.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: Updating smithi192:/etc/ceph/ceph.conf 2023-07-20T18:53:01.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: Updating smithi088:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:53:01.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: Updating smithi192:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:53:01.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: mon.b calling monitor election 2023-07-20T18:53:01.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: mon.c calling monitor election 2023-07-20T18:53:01.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: mon.a calling monitor election 2023-07-20T18:53:01.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T18:53:01.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: monmap e3: 3 mons at {a=[v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2023-07-20T18:53:01.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: fsmap 2023-07-20T18:53:01.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T18:53:01.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: mgrmap e13: x(active, since 57s) 2023-07-20T18:53:01.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T18:53:01.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: Cluster is now healthy 2023-07-20T18:53:01.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: overall HEALTH_OK 2023-07-20T18:53:01.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:01.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: Updating smithi057:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T18:53:01.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:01.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:02.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:02.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:02 smithi057 ceph-mon[126340]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T18:53:02.594 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:53:02.594 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T18:53:02.594 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:02.594 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:02 smithi057 ceph-mon[126340]: Reconfiguring daemon mon.a on smithi057 2023-07-20T18:53:02.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:02.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:02 smithi192 ceph-mon[129200]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T18:53:02.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:53:02.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T18:53:02.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:02.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:02 smithi192 ceph-mon[129200]: Reconfiguring daemon mon.a on smithi057 2023-07-20T18:53:02.860 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:53:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:53:02.552+0000 7f03e68c1700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-07-20T18:53:02.872 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:53:02.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:02 smithi088 ceph-mon[128823]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T18:53:02.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:53:02.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T18:53:02.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:02.873 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:02 smithi088 ceph-mon[128823]: Reconfiguring daemon mon.a on smithi057 2023-07-20T18:53:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.479+0000 7f5c1906e700 1 -- 172.21.15.57:0/3117342084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c14100790 msgr2=0x7f5c14100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.479+0000 7f5c1906e700 1 --2- 172.21.15.57:0/3117342084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c14100790 0x7f5c14100c10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009990 tx=0x7f5c08009ca0 comp rx=0 tx=0).stop 2023-07-20T18:53:03.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.480+0000 7f5c1906e700 1 -- 172.21.15.57:0/3117342084 shutdown_connections 2023-07-20T18:53:03.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.480+0000 7f5c1906e700 1 --2- 172.21.15.57:0/3117342084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c14106790 0x7f5c14106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:03.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.480+0000 7f5c1906e700 1 --2- 172.21.15.57:0/3117342084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c14101150 0x7f5c1410d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:03.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.480+0000 7f5c1906e700 1 --2- 172.21.15.57:0/3117342084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c14100790 0x7f5c14100c10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:03.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.480+0000 7f5c1906e700 1 -- 172.21.15.57:0/3117342084 >> 172.21.15.57:0/3117342084 conn(0x7f5c140fc260 msgr2=0x7f5c140fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:53:03.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.480+0000 7f5c1906e700 1 -- 172.21.15.57:0/3117342084 shutdown_connections 2023-07-20T18:53:03.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.480+0000 7f5c1906e700 1 -- 172.21.15.57:0/3117342084 wait complete. 2023-07-20T18:53:03.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1906e700 1 Processor -- start 2023-07-20T18:53:03.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1906e700 1 -- start start 2023-07-20T18:53:03.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1906e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c14100790 0x7f5c1419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:03.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1906e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c14101150 0x7f5c1419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:03.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1906e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c14106790 0x7f5c141989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:03.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1906e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14199150 con 0x7f5c14106790 2023-07-20T18:53:03.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1906e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c141992c0 con 0x7f5c14101150 2023-07-20T18:53:03.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1906e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14199480 con 0x7f5c14100790 2023-07-20T18:53:03.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c14101150 0x7f5c1419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:03.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c14101150 0x7f5c1419eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:37568/0 (socket says 172.21.15.57:37568) 2023-07-20T18:53:03.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.481+0000 7f5c1259c700 1 -- 172.21.15.57:0/3234411814 learned_addr learned my addr 172.21.15.57:0/3234411814 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:53:03.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5c1359e700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c14106790 0x7f5c141989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:03.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5c12d9d700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c14100790 0x7f5c1419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:03.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5c1259c700 1 -- 172.21.15.57:0/3234411814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c14100790 msgr2=0x7f5c1419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:03.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5c1259c700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c14100790 0x7f5c1419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:03.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5c1259c700 1 -- 172.21.15.57:0/3234411814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c14106790 msgr2=0x7f5c141989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:03.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5c1259c700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c14106790 0x7f5c141989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:03.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5c1259c700 1 -- 172.21.15.57:0/3234411814 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c080095f0 con 0x7f5c14101150 2023-07-20T18:53:03.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5c1259c700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c14101150 0x7f5c1419eef0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5c08005910 tx=0x7f5c08005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:53:03.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5bfbfff700 1 -- 172.21.15.57:0/3234411814 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0801d070 con 0x7f5c14101150 2023-07-20T18:53:03.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5c1906e700 1 -- 172.21.15.57:0/3234411814 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c14199700 con 0x7f5c14101150 2023-07-20T18:53:03.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5bfbfff700 1 -- 172.21.15.57:0/3234411814 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f5c0800fb40 con 0x7f5c14101150 2023-07-20T18:53:03.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.482+0000 7f5bfbfff700 1 -- 172.21.15.57:0/3234411814 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c080173f0 con 0x7f5c14101150 2023-07-20T18:53:03.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.483+0000 7f5c1906e700 1 -- 172.21.15.57:0/3234411814 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c141a5930 con 0x7f5c14101150 2023-07-20T18:53:03.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.484+0000 7f5bfbfff700 1 -- 172.21.15.57:0/3234411814 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f5c0800fcb0 con 0x7f5c14101150 2023-07-20T18:53:03.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.484+0000 7f5c1906e700 1 -- 172.21.15.57:0/3234411814 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1404e470 con 0x7f5c14101150 2023-07-20T18:53:03.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.484+0000 7f5bfbfff700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f5bfc0389e0 0x7f5bfc03aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:03.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.484+0000 7f5bfbfff700 1 -- 172.21.15.57:0/3234411814 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5c0804c6d0 con 0x7f5c14101150 2023-07-20T18:53:03.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.484+0000 7f5c12d9d700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f5bfc0389e0 0x7f5bfc03aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:03.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.485+0000 7f5c12d9d700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f5bfc0389e0 0x7f5bfc03aea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5c04005950 tx=0x7f5c0400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:53:03.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.491+0000 7f5bfbfff700 1 -- 172.21.15.57:0/3234411814 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5c0802a910 con 0x7f5c14101150 2023-07-20T18:53:03.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.701+0000 7f5c1906e700 1 -- 172.21.15.57:0/3234411814 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5c141a5ac0 con 0x7f5c14101150 2023-07-20T18:53:03.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.702+0000 7f5bfbfff700 1 -- 172.21.15.57:0/3234411814 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7f5c08027030 con 0x7f5c14101150 2023-07-20T18:53:03.742 INFO:teuthology.orchestra.run.smithi057.stdout:# minimal ceph.conf for 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:53:03.742 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2023-07-20T18:53:03.742 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T18:53:03.742 INFO:teuthology.orchestra.run.smithi057.stdout: mon_host = [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2023-07-20T18:53:03.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 -- 172.21.15.57:0/3234411814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f5bfc0389e0 msgr2=0x7f5bfc03aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:03.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f5bfc0389e0 0x7f5bfc03aea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5c04005950 tx=0x7f5c0400b410 comp rx=0 tx=0).stop 2023-07-20T18:53:03.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 -- 172.21.15.57:0/3234411814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c14101150 msgr2=0x7f5c1419eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:03.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c14101150 0x7f5c1419eef0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5c08005910 tx=0x7f5c08005740 comp rx=0 tx=0).stop 2023-07-20T18:53:03.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 -- 172.21.15.57:0/3234411814 shutdown_connections 2023-07-20T18:53:03.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f5bfc0389e0 0x7f5bfc03aea0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:03.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c14100790 0x7f5c1419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:03.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c14106790 0x7f5c141989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:03.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 --2- 172.21.15.57:0/3234411814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c14101150 0x7f5c1419eef0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:03.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 -- 172.21.15.57:0/3234411814 >> 172.21.15.57:0/3234411814 conn(0x7f5c140fc260 msgr2=0x7f5c1410bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:53:03.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.704+0000 7f5c1906e700 1 -- 172.21.15.57:0/3234411814 shutdown_connections 2023-07-20T18:53:03.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:03.705+0000 7f5c1906e700 1 -- 172.21.15.57:0/3234411814 wait complete. 2023-07-20T18:53:03.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:03 smithi057 ceph-mon[126340]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:03.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:03.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:03.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:03.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:53:03.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T18:53:03.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:04.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:03 smithi192 ceph-mon[129200]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:04.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:04.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:04.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:04.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:53:04.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T18:53:04.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:04.105 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:03 smithi088 ceph-mon[128823]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:04.105 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:04.105 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:04.105 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:04.106 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:53:04.106 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T18:53:04.106 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:04.214 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-20T18:53:04.215 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:53:04.215 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T18:53:04.290 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:53:04.290 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:53:04.398 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:53:04.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T18:53:04.430 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:53:04.431 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:53:04.496 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:53:04.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T18:53:04.534 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:53:04.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T18:53:04.602 INFO:tasks.cephadm:Adding mgr.x on smithi057 2023-07-20T18:53:04.603 INFO:tasks.cephadm:Adding mgr.y on smithi088 2023-07-20T18:53:04.603 INFO:tasks.cephadm:Adding mgr.z on smithi192 2023-07-20T18:53:04.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch apply mgr '3;smithi057=x;smithi088=y;smithi192=z' 2023-07-20T18:53:04.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:04 smithi192 ceph-mon[129200]: Reconfiguring mon.b (monmap changed)... 2023-07-20T18:53:04.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:04 smithi192 ceph-mon[129200]: Reconfiguring daemon mon.b on smithi088 2023-07-20T18:53:04.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:04 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3234411814' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:04.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:04.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:04.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:53:04.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T18:53:04.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:05.122 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:04 smithi088 ceph-mon[128823]: Reconfiguring mon.b (monmap changed)... 2023-07-20T18:53:05.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:04 smithi088 ceph-mon[128823]: Reconfiguring daemon mon.b on smithi088 2023-07-20T18:53:05.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:04 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3234411814' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:05.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:05.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:05.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:53:05.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T18:53:05.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:05.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:04 smithi057 ceph-mon[126340]: Reconfiguring mon.b (monmap changed)... 2023-07-20T18:53:05.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:04 smithi057 ceph-mon[126340]: Reconfiguring daemon mon.b on smithi088 2023-07-20T18:53:05.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:04 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3234411814' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:05.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:05.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:05.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T18:53:05.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T18:53:05.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:05.308 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:53:05.927 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:05 smithi192 ceph-mon[129200]: Reconfiguring mon.c (monmap changed)... 2023-07-20T18:53:05.927 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:05 smithi192 ceph-mon[129200]: Reconfiguring daemon mon.c on smithi192 2023-07-20T18:53:05.927 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:05 smithi192 ceph-mon[129200]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:05.927 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:05 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:05.927 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:05 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:06.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:05 smithi088 ceph-mon[128823]: Reconfiguring mon.c (monmap changed)... 2023-07-20T18:53:06.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:05 smithi088 ceph-mon[128823]: Reconfiguring daemon mon.c on smithi192 2023-07-20T18:53:06.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:05 smithi088 ceph-mon[128823]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:06.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:05 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:06.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:05 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:06.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:05 smithi057 ceph-mon[126340]: Reconfiguring mon.c (monmap changed)... 2023-07-20T18:53:06.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:05 smithi057 ceph-mon[126340]: Reconfiguring daemon mon.c on smithi192 2023-07-20T18:53:06.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:05 smithi057 ceph-mon[126340]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:06.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:05 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:06.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:05 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:06.726 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.682+0000 7f8fee50d700 1 -- 172.21.15.192:0/2400934040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fe8101a30 msgr2=0x7f8fe8101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:06.727 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.682+0000 7f8fee50d700 1 --2- 172.21.15.192:0/2400934040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fe8101a30 0x7f8fe8101e90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009990 tx=0x7f8fdc009ca0 comp rx=0 tx=0).stop 2023-07-20T18:53:06.727 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 -- 172.21.15.192:0/2400934040 shutdown_connections 2023-07-20T18:53:06.728 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 --2- 172.21.15.192:0/2400934040 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fe81023d0 0x7f8fe8108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:06.728 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 --2- 172.21.15.192:0/2400934040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe8100830 0x7f8fe8100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:06.728 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 --2- 172.21.15.192:0/2400934040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fe8101a30 0x7f8fe8101e90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:06.728 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 -- 172.21.15.192:0/2400934040 >> 172.21.15.192:0/2400934040 conn(0x7f8fe80fbdf0 msgr2=0x7f8fe80fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:53:06.729 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 -- 172.21.15.192:0/2400934040 shutdown_connections 2023-07-20T18:53:06.729 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 -- 172.21.15.192:0/2400934040 wait complete. 2023-07-20T18:53:06.729 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 Processor -- start 2023-07-20T18:53:06.729 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 -- start start 2023-07-20T18:53:06.729 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe8100830 0x7f8fe8196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:06.729 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fe8101a30 0x7f8fe8196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:06.729 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.683+0000 7f8fee50d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fe81023d0 0x7f8fe819db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:06.730 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fee50d700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe819e0d0 con 0x7f8fe8100830 2023-07-20T18:53:06.730 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fee50d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe819e210 con 0x7f8fe8101a30 2023-07-20T18:53:06.730 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fee50d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe819e3a0 con 0x7f8fe81023d0 2023-07-20T18:53:06.730 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fecaaa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fe81023d0 0x7f8fe819db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:06.730 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fecaaa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fe81023d0 0x7f8fe819db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:48102/0 (socket says 172.21.15.192:48102) 2023-07-20T18:53:06.730 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fecaaa700 1 -- 172.21.15.192:0/1661159444 learned_addr learned my addr 172.21.15.192:0/1661159444 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-07-20T18:53:06.731 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fe7fff700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe8100830 0x7f8fe8196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:06.731 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fe77fe700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fe8101a30 0x7f8fe8196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:06.731 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fecaaa700 1 -- 172.21.15.192:0/1661159444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fe8101a30 msgr2=0x7f8fe8196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:06.731 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fecaaa700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fe8101a30 0x7f8fe8196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:06.731 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fecaaa700 1 -- 172.21.15.192:0/1661159444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe8100830 msgr2=0x7f8fe8196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:06.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fecaaa700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe8100830 0x7f8fe8196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:06.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fecaaa700 1 -- 172.21.15.192:0/1661159444 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc0095f0 con 0x7f8fe81023d0 2023-07-20T18:53:06.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fecaaa700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fe81023d0 0x7f8fe819db00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000b9a0 tx=0x7f8fe000bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:53:06.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fe57fa700 1 -- 172.21.15.192:0/1661159444 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe000c9b0 con 0x7f8fe81023d0 2023-07-20T18:53:06.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fee50d700 1 -- 172.21.15.192:0/1661159444 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe819e680 con 0x7f8fe81023d0 2023-07-20T18:53:06.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fe57fa700 1 -- 172.21.15.192:0/1661159444 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f8fe0014440 con 0x7f8fe81023d0 2023-07-20T18:53:06.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fe57fa700 1 -- 172.21.15.192:0/1661159444 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe00075d0 con 0x7f8fe81023d0 2023-07-20T18:53:06.733 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.684+0000 7f8fee50d700 1 -- 172.21.15.192:0/1661159444 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe819ebb0 con 0x7f8fe81023d0 2023-07-20T18:53:06.734 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.685+0000 7f8fee50d700 1 -- 172.21.15.192:0/1661159444 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe8107cf0 con 0x7f8fe81023d0 2023-07-20T18:53:06.734 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.685+0000 7f8fe57fa700 1 -- 172.21.15.192:0/1661159444 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f8fe0014920 con 0x7f8fe81023d0 2023-07-20T18:53:06.734 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.685+0000 7f8fe57fa700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8fd00389e0 0x7f8fd003aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:06.735 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.685+0000 7f8fe57fa700 1 -- 172.21.15.192:0/1661159444 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8fe0006e40 con 0x7f8fe81023d0 2023-07-20T18:53:06.736 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.686+0000 7f8fe7fff700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8fd00389e0 0x7f8fd003aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:06.736 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.687+0000 7f8fe7fff700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8fd00389e0 0x7f8fd003aea0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8005fd0 tx=0x7f8fd8005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:53:06.737 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.690+0000 7f8fe57fa700 1 -- 172.21.15.192:0/1661159444 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8fe0007730 con 0x7f8fe81023d0 2023-07-20T18:53:06.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:06.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:06.830 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:06.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:06.925 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.884+0000 7f8fee50d700 1 -- 172.21.15.192:0/1661159444 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi057=x;smithi088=y;smithi192=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fe8060c40 con 0x7f8fd00389e0 2023-07-20T18:53:06.936 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.895+0000 7f8fe57fa700 1 -- 172.21.15.192:0/1661159444 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8fe8060c40 con 0x7f8fd00389e0 2023-07-20T18:53:06.936 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mgr update... 2023-07-20T18:53:06.938 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.897+0000 7f8fee50d700 1 -- 172.21.15.192:0/1661159444 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8fd00389e0 msgr2=0x7f8fd003aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:06.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.897+0000 7f8fee50d700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8fd00389e0 0x7f8fd003aea0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8005fd0 tx=0x7f8fd8005e50 comp rx=0 tx=0).stop 2023-07-20T18:53:06.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.898+0000 7f8fee50d700 1 -- 172.21.15.192:0/1661159444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fe81023d0 msgr2=0x7f8fe819db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:06.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.898+0000 7f8fee50d700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fe81023d0 0x7f8fe819db00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000b9a0 tx=0x7f8fe000bd60 comp rx=0 tx=0).stop 2023-07-20T18:53:06.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.898+0000 7f8fee50d700 1 -- 172.21.15.192:0/1661159444 shutdown_connections 2023-07-20T18:53:06.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.898+0000 7f8fee50d700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8fd00389e0 0x7f8fd003aea0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:06.939 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.898+0000 7f8fee50d700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fe81023d0 0x7f8fe819db00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:06.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.898+0000 7f8fee50d700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe8100830 0x7f8fe8196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:06.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.898+0000 7f8fee50d700 1 --2- 172.21.15.192:0/1661159444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fe8101a30 0x7f8fe8196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:06.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.898+0000 7f8fee50d700 1 -- 172.21.15.192:0/1661159444 >> 172.21.15.192:0/1661159444 conn(0x7f8fe80fbdf0 msgr2=0x7f8fe8104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:53:06.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.898+0000 7f8fee50d700 1 -- 172.21.15.192:0/1661159444 shutdown_connections 2023-07-20T18:53:06.940 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:53:06.898+0000 7f8fee50d700 1 -- 172.21.15.192:0/1661159444 wait complete. 2023-07-20T18:53:07.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:07.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:07.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:07.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:07.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:07.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:07.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:07.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:07.374 DEBUG:teuthology.orchestra.run.smithi088:mgr.y> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.y.service 2023-07-20T18:53:07.377 DEBUG:teuthology.orchestra.run.smithi192:mgr.z> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.z.service 2023-07-20T18:53:07.379 INFO:tasks.cephadm:Deploying OSDs... 2023-07-20T18:53:07.379 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:53:07.380 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T18:53:07.408 INFO:journalctl@ceph.mgr.y.smithi088.stdout:-- Logs begin at Thu 2023-07-20 18:25:07 UTC. -- 2023-07-20T18:53:07.409 INFO:journalctl@ceph.mgr.z.smithi192.stdout:-- Logs begin at Thu 2023-07-20 18:24:33 UTC. -- 2023-07-20T18:53:07.423 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T18:53:07.423 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 2023-07-20T18:53:07.510 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T18:53:07.511 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:07.511 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 446 Links: 1 2023-07-20T18:53:07.512 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:07.512 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:07.512 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-07-20 18:52:14.370761324 +0000 2023-07-20T18:53:07.512 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-07-20 18:45:00.665847066 +0000 2023-07-20T18:53:07.512 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-07-20 18:45:00.665847066 +0000 2023-07-20T18:53:07.513 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-07-20 18:45:00.635847486 +0000 2023-07-20T18:53:07.513 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T18:53:07.620 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-07-20T18:53:07.620 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-07-20T18:53:07.620 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000385767 s, 1.3 MB/s 2023-07-20T18:53:07.622 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T18:53:07.670 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 2023-07-20T18:53:07.757 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T18:53:07.757 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:07.757 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 452 Links: 1 2023-07-20T18:53:07.758 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:07.758 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:07.758 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-07-20 18:52:14.370761324 +0000 2023-07-20T18:53:07.758 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-07-20 18:45:01.513835204 +0000 2023-07-20T18:53:07.758 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-07-20 18:45:01.513835204 +0000 2023-07-20T18:53:07.758 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-07-20 18:45:01.480835665 +0000 2023-07-20T18:53:07.759 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T18:53:07.865 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-07-20T18:53:07.865 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-07-20T18:53:07.865 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000361824 s, 1.4 MB/s 2023-07-20T18:53:07.867 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T18:53:07.912 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 2023-07-20T18:53:07.997 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T18:53:07.998 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:07.998 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 458 Links: 1 2023-07-20T18:53:07.998 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:07.998 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:07.998 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-07-20 18:52:14.370761324 +0000 2023-07-20T18:53:07.999 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-07-20 18:45:02.370823215 +0000 2023-07-20T18:53:07.999 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-07-20 18:45:02.370823215 +0000 2023-07-20T18:53:07.999 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-07-20 18:45:02.339823649 +0000 2023-07-20T18:53:07.999 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T18:53:08.103 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-07-20T18:53:08.104 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-07-20T18:53:08.104 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000618302 s, 828 kB/s 2023-07-20T18:53:08.106 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T18:53:08.151 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 2023-07-20T18:53:08.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:08.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi057=x;smithi088=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:53:08.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: Saving service mgr spec with placement smithi057=x;smithi088=y;smithi192=z;count:3 2023-07-20T18:53:08.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:08.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:08.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:08.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:08.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:08.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T18:53:08.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T18:53:08.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T18:53:08.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:08.235 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T18:53:08.236 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:08.236 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 464 Links: 1 2023-07-20T18:53:08.236 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:08.236 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:08.237 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-07-20 18:52:14.370761324 +0000 2023-07-20T18:53:08.237 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-07-20 18:45:03.199811619 +0000 2023-07-20T18:53:08.237 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-07-20 18:45:03.199811619 +0000 2023-07-20T18:53:08.237 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-07-20 18:45:03.165812094 +0000 2023-07-20T18:53:08.238 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T18:53:08.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:08.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi057=x;smithi088=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:53:08.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: Saving service mgr spec with placement smithi057=x;smithi088=y;smithi192=z;count:3 2023-07-20T18:53:08.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:08.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:08.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:08.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:08.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:08.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T18:53:08.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T18:53:08.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T18:53:08.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:08.341 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-07-20T18:53:08.341 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-07-20T18:53:08.342 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000345391 s, 1.5 MB/s 2023-07-20T18:53:08.344 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T18:53:08.373 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:08.373 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi057=x;smithi088=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:53:08.373 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: Saving service mgr spec with placement smithi057=x;smithi088=y;smithi192=z;count:3 2023-07-20T18:53:08.373 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:08.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:08.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:08.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:08.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:08.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T18:53:08.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T18:53:08.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T18:53:08.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:08.388 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-20T18:53:08.389 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T18:53:08.409 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T18:53:08.409 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2023-07-20T18:53:08.469 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T18:53:08.469 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:08.470 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 380052 Links: 1 2023-07-20T18:53:08.470 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:08.470 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:08.470 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-20 18:52:52.997559527 +0000 2023-07-20T18:53:08.470 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-20 18:41:35.158234632 +0000 2023-07-20T18:53:08.471 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-20 18:41:35.158234632 +0000 2023-07-20T18:53:08.471 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-20T18:53:08.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T18:53:08.537 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-20T18:53:08.537 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-20T18:53:08.537 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000247538 s, 2.1 MB/s 2023-07-20T18:53:08.538 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T18:53:08.597 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2023-07-20T18:53:08.655 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T18:53:08.655 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:08.655 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 381367 Links: 1 2023-07-20T18:53:08.655 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:08.656 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:08.656 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-20 18:52:52.997559527 +0000 2023-07-20T18:53:08.656 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-20 18:41:35.704224131 +0000 2023-07-20T18:53:08.656 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-20 18:41:35.704224131 +0000 2023-07-20T18:53:08.656 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-20T18:53:08.656 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T18:53:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-20T18:53:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-20T18:53:08.722 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.00022792 s, 2.2 MB/s 2023-07-20T18:53:08.722 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T18:53:08.781 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2023-07-20T18:53:08.839 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T18:53:08.839 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:08.839 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 374332 Links: 1 2023-07-20T18:53:08.839 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:08.839 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:08.839 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-20 18:52:52.997559527 +0000 2023-07-20T18:53:08.840 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-20 18:41:36.233213957 +0000 2023-07-20T18:53:08.840 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-20 18:41:36.233213957 +0000 2023-07-20T18:53:08.840 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-20T18:53:08.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T18:53:08.905 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-20T18:53:08.905 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-20T18:53:08.906 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000253365 s, 2.0 MB/s 2023-07-20T18:53:08.906 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T18:53:08.965 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2023-07-20T18:53:09.015 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:08 smithi192 ceph-mon[129200]: Deploying daemon mgr.z on smithi192 2023-07-20T18:53:09.022 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T18:53:09.023 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:09.023 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 380257 Links: 1 2023-07-20T18:53:09.023 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:09.023 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:09.024 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-20 18:52:52.997559527 +0000 2023-07-20T18:53:09.024 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-20 18:41:36.783203379 +0000 2023-07-20T18:53:09.024 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-20 18:41:36.783203379 +0000 2023-07-20T18:53:09.024 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-20T18:53:09.025 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T18:53:09.089 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-20T18:53:09.089 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-20T18:53:09.089 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000315111 s, 1.6 MB/s 2023-07-20T18:53:09.090 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T18:53:09.147 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-20T18:53:09.148 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T18:53:09.173 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T18:53:09.174 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2023-07-20T18:53:09.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:08 smithi057 ceph-mon[126340]: Deploying daemon mgr.z on smithi192 2023-07-20T18:53:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T18:53:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:09.233 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 380482 Links: 1 2023-07-20T18:53:09.233 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:09.233 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:09.234 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-07-20 18:53:09.147781879 +0000 2023-07-20T18:53:09.234 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-07-20 18:41:35.148146211 +0000 2023-07-20T18:53:09.235 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-07-20 18:41:35.148146211 +0000 2023-07-20T18:53:09.235 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-07-20T18:53:09.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T18:53:09.300 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:09 smithi192 systemd[1]: Starting Ceph mgr.z for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:53:09.303 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-07-20T18:53:09.304 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-07-20T18:53:09.304 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000222467 s, 2.3 MB/s 2023-07-20T18:53:09.306 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T18:53:09.365 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2023-07-20T18:53:09.373 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:08 smithi088 ceph-mon[128823]: Deploying daemon mgr.z on smithi192 2023-07-20T18:53:09.425 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T18:53:09.425 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:09.426 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 381185 Links: 1 2023-07-20T18:53:09.426 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:09.427 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:09.427 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-07-20 18:53:09.147781879 +0000 2023-07-20T18:53:09.427 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-07-20 18:41:35.668136154 +0000 2023-07-20T18:53:09.428 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-07-20 18:41:35.668136154 +0000 2023-07-20T18:53:09.428 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-07-20T18:53:09.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T18:53:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-07-20T18:53:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-07-20T18:53:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000200905 s, 2.5 MB/s 2023-07-20T18:53:09.493 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T18:53:09.551 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2023-07-20T18:53:09.580 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:09 smithi192 podman[131139]: 2023-07-20T18:53:09.598 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T18:53:09.598 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:09.598 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 381331 Links: 1 2023-07-20T18:53:09.598 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:09.599 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:09.599 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-07-20 18:53:09.147781879 +0000 2023-07-20T18:53:09.599 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-07-20 18:41:36.175126349 +0000 2023-07-20T18:53:09.599 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-07-20 18:41:36.175126349 +0000 2023-07-20T18:53:09.599 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-07-20T18:53:09.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T18:53:09.665 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-07-20T18:53:09.665 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-07-20T18:53:09.665 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000228667 s, 2.2 MB/s 2023-07-20T18:53:09.666 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T18:53:09.724 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2023-07-20T18:53:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T18:53:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T18:53:09.781 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 381486 Links: 1 2023-07-20T18:53:09.781 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T18:53:09.781 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T18:53:09.781 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-07-20 18:53:09.147781879 +0000 2023-07-20T18:53:09.781 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-07-20 18:41:36.741115403 +0000 2023-07-20T18:53:09.782 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-07-20 18:41:36.741115403 +0000 2023-07-20T18:53:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-07-20T18:53:09.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T18:53:09.849 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-07-20T18:53:09.849 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-07-20T18:53:09.850 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000195305 s, 2.6 MB/s 2023-07-20T18:53:09.850 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T18:53:09.909 INFO:tasks.cephadm:Deploying osd.0 on smithi057 with /dev/vg_nvme/lv_4... 2023-07-20T18:53:09.909 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T18:53:09.939 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-mgr[131323]: -- 172.21.15.192:0/2753107991 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56488d0811e0 con 0x56488d056800 2023-07-20T18:53:09.939 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:09 smithi192 bash[131139]: 6f8fd40071777d12ca6d2c6e34f56857826a99335237995b1448d0576bd443a8 2023-07-20T18:53:09.939 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:09 smithi192 systemd[1]: Started Ceph mgr.z for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:53:10.200 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-mon[129200]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:10.201 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:10.201 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:10.201 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:10.201 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T18:53:10.201 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T18:53:10.201 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T18:53:10.202 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:10.202 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:09.897+0000 7fddde2a8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T18:53:10.202 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:09 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:09.978+0000 7fddde2a8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T18:53:10.202 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:10.067+0000 7fddde2a8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T18:53:10.202 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:10.159+0000 7fddde2a8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T18:53:10.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:09 smithi057 ceph-mon[126340]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:10.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:10.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:10.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:10.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T18:53:10.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T18:53:10.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T18:53:10.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:10.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:09 smithi088 ceph-mon[128823]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:10.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:10.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:10.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:10.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T18:53:10.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T18:53:10.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T18:53:10.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:10.580 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:10.328+0000 7fddde2a8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T18:53:10.645 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:53:10.961 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:10.637+0000 7fddde2a8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T18:53:10.961 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:10.725+0000 7fddde2a8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T18:53:10.961 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:10.853+0000 7fddde2a8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T18:53:11.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:10 smithi088 ceph-mon[128823]: Deploying daemon mgr.y on smithi088 2023-07-20T18:53:11.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:10 smithi057 ceph-mon[126340]: Deploying daemon mgr.y on smithi088 2023-07-20T18:53:11.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:10 smithi192 ceph-mon[129200]: Deploying daemon mgr.y on smithi088 2023-07-20T18:53:11.330 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:10.948+0000 7fddde2a8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T18:53:11.623 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:11 smithi088 systemd[1]: Starting Ceph mgr.y for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:53:11.830 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:11.368+0000 7fddde2a8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T18:53:11.961 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:11 smithi088 podman[130287]: 2023-07-20T18:53:11.961 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:11 smithi088 bash[130287]: 1d2da0bd749258c29f1795b35862557521149cbff9293ccadb03ac68db5df26c 2023-07-20T18:53:11.961 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:11 smithi088 ceph-mgr[130327]: -- 172.21.15.88:0/402716529 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bb3dd071e0 con 0x55bb3dcdd000 2023-07-20T18:53:12.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:11 smithi057 ceph-mon[126340]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:12.275 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:11 smithi088 ceph-mon[128823]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:12.276 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:11 smithi088 systemd[1]: Started Ceph mgr.y for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:53:12.276 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:11 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:11.966+0000 7fe2dcd27200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T18:53:12.276 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:12.048+0000 7fe2dcd27200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T18:53:12.276 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:12.136+0000 7fe2dcd27200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T18:53:12.277 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:12.235+0000 7fe2dcd27200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T18:53:12.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:11 smithi192 ceph-mon[129200]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:12.330 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:11.945+0000 7fddde2a8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T18:53:12.331 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:12.116+0000 7fddde2a8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T18:53:12.623 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:12.417+0000 7fe2dcd27200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T18:53:12.686 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T18:53:12.686 INFO:teuthology.orchestra.run.smithi057.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T18:53:12.687 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10+0 records in 2023-07-20T18:53:12.687 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2023-07-20T18:53:12.687 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0311185 s, 337 MB/s 2023-07-20T18:53:12.687 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping successful for: 2023-07-20T18:53:13.110 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:12.746+0000 7fe2dcd27200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T18:53:13.111 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:12.834+0000 7fe2dcd27200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T18:53:13.111 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:12.967+0000 7fe2dcd27200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T18:53:13.111 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:13 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:13.070+0000 7fe2dcd27200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T18:53:13.111 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.111 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.112 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.112 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.112 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:13.112 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-mon[128823]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:13.112 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:12 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.128 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi057:vg_nvme/lv_4 2023-07-20T18:53:13.243 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:12.929+0000 7fddde2a8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T18:53:13.244 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:13.026+0000 7fddde2a8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T18:53:13.244 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:12 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.244 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:12 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.244 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:12 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.245 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:12 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.245 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:12 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:13.245 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:12 smithi192 ceph-mon[129200]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:13.245 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:12 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.476 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:12 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.476 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:12 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:12 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:12 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:12 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:13.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:12 smithi057 ceph-mon[126340]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:13.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:12 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:13.547 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:13.202+0000 7fddde2a8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T18:53:13.547 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:13.505+0000 7fddde2a8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T18:53:13.832 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:13.743+0000 7fddde2a8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T18:53:13.859 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:53:13.873 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:13 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:13.514+0000 7fe2dcd27200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T18:53:14.200 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:13.891+0000 7fddde2a8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T18:53:14.580 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:14.157+0000 7fddde2a8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T18:53:14.623 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:14.107+0000 7fe2dcd27200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T18:53:14.623 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:14.274+0000 7fe2dcd27200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T18:53:14.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:14.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:14.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:14.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:14.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:14.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:14.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:14.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T18:53:14.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T18:53:14.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:15.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:15.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:15.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T18:53:15.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T18:53:15.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:15.080 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:14.858+0000 7fddde2a8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T18:53:15.080 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:14.987+0000 7fddde2a8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T18:53:15.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:15.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:15.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T18:53:15.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T18:53:15.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:15.336 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:15.033+0000 7fe2dcd27200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T18:53:15.337 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:15.125+0000 7fe2dcd27200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T18:53:15.549 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:15.178+0000 7fddde2a8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T18:53:15.549 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:15.292+0000 7fddde2a8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T18:53:15.817 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:15.507+0000 7fddde2a8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T18:53:15.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.787+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3364886898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cc00ff110 msgr2=0x7f2cc00ff530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:15.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.787+0000 7f2cc4cb2700 1 --2- 172.21.15.57:0/3364886898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cc00ff110 0x7f2cc00ff530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009990 tx=0x7f2cb0009ca0 comp rx=0 tx=0).stop 2023-07-20T18:53:15.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.787+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3364886898 shutdown_connections 2023-07-20T18:53:15.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.787+0000 7f2cc4cb2700 1 --2- 172.21.15.57:0/3364886898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc0100430 0x7f2cc01008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:15.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.787+0000 7f2cc4cb2700 1 --2- 172.21.15.57:0/3364886898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cc00ff110 0x7f2cc00ff530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:15.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.787+0000 7f2cc4cb2700 1 --2- 172.21.15.57:0/3364886898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cc00ffa70 0x7f2cc00ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:15.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.787+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3364886898 >> 172.21.15.57:0/3364886898 conn(0x7f2cc00facf0 msgr2=0x7f2cc00fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:53:15.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.787+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3364886898 shutdown_connections 2023-07-20T18:53:15.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.787+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3364886898 wait complete. 2023-07-20T18:53:15.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.789+0000 7f2cc4cb2700 1 Processor -- start 2023-07-20T18:53:15.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.789+0000 7f2cc4cb2700 1 -- start start 2023-07-20T18:53:15.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cc4cb2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc00ff110 0x7f2cc01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:15.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cc4cb2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cc00ffa70 0x7f2cc0198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:15.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cc4cb2700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cc0100430 0x7f2cc019fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:15.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cc4cb2700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc01a0360 con 0x7f2cc0100430 2023-07-20T18:53:15.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cc4cb2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc01a04a0 con 0x7f2cc00ffa70 2023-07-20T18:53:15.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cc4cb2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc01a0630 con 0x7f2cc00ff110 2023-07-20T18:53:15.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cbe59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc00ff110 0x7f2cc01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:15.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cbe59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc00ff110 0x7f2cc01987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:37708/0 (socket says 172.21.15.57:37708) 2023-07-20T18:53:15.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cbe59c700 1 -- 172.21.15.57:0/3486333953 learned_addr learned my addr 172.21.15.57:0/3486333953 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:53:15.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cbdd9b700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cc00ffa70 0x7f2cc0198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:15.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cbed9d700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cc0100430 0x7f2cc019fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:15.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cbdd9b700 1 -- 172.21.15.57:0/3486333953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc00ff110 msgr2=0x7f2cc01987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:15.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cbdd9b700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc00ff110 0x7f2cc01987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:15.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cbdd9b700 1 -- 172.21.15.57:0/3486333953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cc0100430 msgr2=0x7f2cc019fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:15.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cbdd9b700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cc0100430 0x7f2cc019fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:15.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.790+0000 7f2cbdd9b700 1 -- 172.21.15.57:0/3486333953 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb00095f0 con 0x7f2cc00ffa70 2023-07-20T18:53:15.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.791+0000 7f2cbdd9b700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cc00ffa70 0x7f2cc0198d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400e9f0 tx=0x7f2cb400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:53:15.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.791+0000 7f2caf7fe700 1 -- 172.21.15.57:0/3486333953 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb400c9c0 con 0x7f2cc00ffa70 2023-07-20T18:53:15.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.791+0000 7f2caf7fe700 1 -- 172.21.15.57:0/3486333953 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f2cb4010430 con 0x7f2cc00ffa70 2023-07-20T18:53:15.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.791+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3486333953 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc01a0910 con 0x7f2cc00ffa70 2023-07-20T18:53:15.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.791+0000 7f2caf7fe700 1 -- 172.21.15.57:0/3486333953 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4018680 con 0x7f2cc00ffa70 2023-07-20T18:53:15.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.791+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3486333953 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc01a0e60 con 0x7f2cc00ffa70 2023-07-20T18:53:15.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.792+0000 7f2caf7fe700 1 -- 172.21.15.57:0/3486333953 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f2cb40187e0 con 0x7f2cc00ffa70 2023-07-20T18:53:15.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.792+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3486333953 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc0109ef0 con 0x7f2cc00ffa70 2023-07-20T18:53:15.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.793+0000 7f2caf7fe700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ca80389e0 0x7f2ca803aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:15.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.793+0000 7f2caf7fe700 1 -- 172.21.15.57:0/3486333953 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2cb4014070 con 0x7f2cc00ffa70 2023-07-20T18:53:15.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.793+0000 7f2cbe59c700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ca80389e0 0x7f2ca803aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:15.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.794+0000 7f2cbe59c700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ca80389e0 0x7f2ca803aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000b2e0 tx=0x7f2cb0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:53:15.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:15.798+0000 7f2caf7fe700 1 -- 172.21.15.57:0/3486333953 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2cb401fd50 con 0x7f2cc00ffa70 2023-07-20T18:53:15.873 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:15.296+0000 7fe2dcd27200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T18:53:15.873 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:15.570+0000 7fe2dcd27200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T18:53:15.873 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:15.803+0000 7fe2dcd27200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T18:53:15.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-mon[128823]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T18:53:15.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-mon[128823]: Reconfiguring daemon mgr.x on smithi057 2023-07-20T18:53:15.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-mon[128823]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:15.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:15.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:15.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:15.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:15.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:16.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:16.015+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3486333953 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cc0060bb0 con 0x7f2ca80389e0 2023-07-20T18:53:16.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-mon[129200]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T18:53:16.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-mon[129200]: Reconfiguring daemon mgr.x on smithi057 2023-07-20T18:53:16.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-mon[129200]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:16.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:16.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:16.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:16.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:16.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:16.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:16.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:15 smithi057 ceph-mon[126340]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T18:53:16.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:15 smithi057 ceph-mon[126340]: Reconfiguring daemon mgr.x on smithi057 2023-07-20T18:53:16.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:15 smithi057 ceph-mon[126340]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:16.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:16.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:16.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:16.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:16.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:16.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:16.253 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:15.954+0000 7fe2dcd27200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T18:53:16.623 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:16 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:16.212+0000 7fe2dcd27200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T18:53:16.824 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:53:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z[131234]: 2023-07-20T18:53:16.372+0000 7fddde2a8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T18:53:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:16.788+0000 7f2caf7fe700 1 -- 172.21.15.57:0/3486333953 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7f2cb4017750 con 0x7f2cc00ffa70 2023-07-20T18:53:17.076 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:16 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:16.904+0000 7fe2dcd27200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T18:53:17.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:16 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:53:17.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:16 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:53:17.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:16 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:17.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:16 smithi088 ceph-mon[128823]: Standby manager daemon z started 2023-07-20T18:53:17.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:16 smithi088 ceph-mon[128823]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T18:53:17.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:16 smithi088 ceph-mon[128823]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T18:53:17.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:16 smithi088 ceph-mon[128823]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T18:53:17.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:16 smithi088 ceph-mon[128823]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T18:53:17.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:16 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:53:17.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:16 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:53:17.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:16 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:17.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:16 smithi192 ceph-mon[129200]: Standby manager daemon z started 2023-07-20T18:53:17.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:16 smithi192 ceph-mon[129200]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T18:53:17.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:16 smithi192 ceph-mon[129200]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T18:53:17.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:16 smithi192 ceph-mon[129200]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T18:53:17.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:16 smithi192 ceph-mon[129200]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T18:53:17.122 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:16 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:53:17.123 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:16 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:53:17.123 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:16 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:17.123 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:16 smithi057 ceph-mon[126340]: Standby manager daemon z started 2023-07-20T18:53:17.123 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:16 smithi057 ceph-mon[126340]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T18:53:17.123 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:16 smithi057 ceph-mon[126340]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T18:53:17.123 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:16 smithi057 ceph-mon[126340]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T18:53:17.123 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:16 smithi057 ceph-mon[126340]: from='mgr.? 172.21.15.192:0/3588292021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T18:53:17.373 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:17 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:17.035+0000 7fe2dcd27200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T18:53:17.373 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:17 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:17.222+0000 7fe2dcd27200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T18:53:17.827 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:17 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:17.334+0000 7fe2dcd27200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T18:53:17.827 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:17 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:17.547+0000 7fe2dcd27200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T18:53:17.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:17 smithi057 ceph-mon[126340]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:53:17.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:17 smithi057 ceph-mon[126340]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:17.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:17 smithi057 ceph-mon[126340]: mgrmap e14: x(active, since 74s), standbys: z 2023-07-20T18:53:17.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T18:53:18.080 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:17 smithi192 ceph-mon[129200]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:53:18.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:17 smithi192 ceph-mon[129200]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:18.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:17 smithi192 ceph-mon[129200]: mgrmap e14: x(active, since 74s), standbys: z 2023-07-20T18:53:18.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T18:53:18.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:17 smithi088 ceph-mon[128823]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:53:18.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:17 smithi088 ceph-mon[128823]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:18.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:17 smithi088 ceph-mon[128823]: mgrmap e14: x(active, since 74s), standbys: z 2023-07-20T18:53:18.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T18:53:18.829 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:53:18 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y[130303]: 2023-07-20T18:53:18.391+0000 7fe2dcd27200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T18:53:18.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:18.799+0000 7f2caf7fe700 1 -- 172.21.15.57:0/3486333953 <== mon.2 v2:172.21.15.88:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f2cb407d4c0 con 0x7f2cc00ffa70 2023-07-20T18:53:19.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:18 smithi192 ceph-mon[129200]: Standby manager daemon y started 2023-07-20T18:53:19.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:18 smithi192 ceph-mon[129200]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T18:53:19.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:18 smithi192 ceph-mon[129200]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T18:53:19.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:18 smithi192 ceph-mon[129200]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T18:53:19.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:18 smithi192 ceph-mon[129200]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T18:53:19.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:18 smithi088 ceph-mon[128823]: Standby manager daemon y started 2023-07-20T18:53:19.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:18 smithi088 ceph-mon[128823]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T18:53:19.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:18 smithi088 ceph-mon[128823]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T18:53:19.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:18 smithi088 ceph-mon[128823]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T18:53:19.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:18 smithi088 ceph-mon[128823]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T18:53:19.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:18 smithi057 ceph-mon[126340]: Standby manager daemon y started 2023-07-20T18:53:19.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:18 smithi057 ceph-mon[126340]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T18:53:19.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:18 smithi057 ceph-mon[126340]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T18:53:19.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:18 smithi057 ceph-mon[126340]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T18:53:19.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:18 smithi057 ceph-mon[126340]: from='mgr.? 172.21.15.88:0/1480848132' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T18:53:20.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:19 smithi088 ceph-mon[128823]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:20.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:19 smithi088 ceph-mon[128823]: mgrmap e15: x(active, since 76s), standbys: z, y 2023-07-20T18:53:20.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:19 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T18:53:20.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:19 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2524995316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "627a67ea-1366-4b3f-babd-067f75bad1e6"}]: dispatch 2023-07-20T18:53:20.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:19 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2524995316' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "627a67ea-1366-4b3f-babd-067f75bad1e6"}]': finished 2023-07-20T18:53:20.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:19 smithi088 ceph-mon[128823]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T18:53:20.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:19 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:20.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:19 smithi057 ceph-mon[126340]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:20.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:19 smithi057 ceph-mon[126340]: mgrmap e15: x(active, since 76s), standbys: z, y 2023-07-20T18:53:20.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:19 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T18:53:20.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:19 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2524995316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "627a67ea-1366-4b3f-babd-067f75bad1e6"}]: dispatch 2023-07-20T18:53:20.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:19 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2524995316' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "627a67ea-1366-4b3f-babd-067f75bad1e6"}]': finished 2023-07-20T18:53:20.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:19 smithi057 ceph-mon[126340]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T18:53:20.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:19 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:20.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:19 smithi192 ceph-mon[129200]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:20.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:19 smithi192 ceph-mon[129200]: mgrmap e15: x(active, since 76s), standbys: z, y 2023-07-20T18:53:20.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:19 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T18:53:20.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:19 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2524995316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "627a67ea-1366-4b3f-babd-067f75bad1e6"}]: dispatch 2023-07-20T18:53:20.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:19 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2524995316' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "627a67ea-1366-4b3f-babd-067f75bad1e6"}]': finished 2023-07-20T18:53:20.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:19 smithi192 ceph-mon[129200]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T18:53:20.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:19 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:21.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:20 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1056786534' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:53:21.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:20 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1056786534' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:53:21.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:20 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1056786534' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:53:22.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:21 smithi088 ceph-mon[128823]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:22.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:21 smithi057 ceph-mon[126340]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:22.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:21 smithi192 ceph-mon[129200]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:24.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:23 smithi088 ceph-mon[128823]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:24.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:23 smithi192 ceph-mon[129200]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:24.551 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:23 smithi057 ceph-mon[126340]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:25.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:25 smithi057 ceph-mon[126340]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:26.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:25 smithi088 ceph-mon[128823]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:26.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:25 smithi192 ceph-mon[129200]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:26.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:26 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T18:53:26.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:26 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:27.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:26 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T18:53:27.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:26 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:27.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:26 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T18:53:27.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:26 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:27.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:27 smithi057 ceph-mon[126340]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:27.973 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:27 smithi057 ceph-mon[126340]: Deploying daemon osd.0 on smithi057 2023-07-20T18:53:28.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:27 smithi088 ceph-mon[128823]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:28.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:27 smithi088 ceph-mon[128823]: Deploying daemon osd.0 on smithi057 2023-07-20T18:53:28.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:27 smithi192 ceph-mon[129200]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:28.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:27 smithi192 ceph-mon[129200]: Deploying daemon osd.0 on smithi057 2023-07-20T18:53:29.933 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:29 smithi057 ceph-mon[126340]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:30.123 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:29 smithi088 ceph-mon[128823]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:30.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:29 smithi192 ceph-mon[129200]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:32.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:31 smithi088 ceph-mon[128823]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:32.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:31 smithi057 ceph-mon[126340]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:32.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:31 smithi192 ceph-mon[129200]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:33.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:33 smithi057 ceph-mon[126340]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:33.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:33.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:33.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:34.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:33 smithi192 ceph-mon[129200]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:34.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:34.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:34.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:34.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:33 smithi088 ceph-mon[128823]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:34.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:34.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:34.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:36.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:35 smithi057 ceph-mon[126340]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:36.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:35 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:36.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:35 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:36.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:35 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:36.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:35 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:36.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:35 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:36.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:35 smithi192 ceph-mon[129200]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:36.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:35 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:36.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:35 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:36.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:35 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:36.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:35 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:36.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:35 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:36.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:35 smithi088 ceph-mon[128823]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:36.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:35 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:36.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:35 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:36.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:35 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:36.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:35 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:36.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:35 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:37.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.124+0000 7f2caf7fe700 1 -- 172.21.15.57:0/3486333953 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2cc0060bb0 con 0x7f2ca80389e0 2023-07-20T18:53:37.166 INFO:teuthology.orchestra.run.smithi057.stdout:Created osd(s) 0 on host 'smithi057' 2023-07-20T18:53:37.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.126+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3486333953 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ca80389e0 msgr2=0x7f2ca803aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:37.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ca80389e0 0x7f2ca803aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000b2e0 tx=0x7f2cb0005fb0 comp rx=0 tx=0).stop 2023-07-20T18:53:37.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3486333953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cc00ffa70 msgr2=0x7f2cc0198d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:37.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cc00ffa70 0x7f2cc0198d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400e9f0 tx=0x7f2cb400ed00 comp rx=0 tx=0).stop 2023-07-20T18:53:37.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3486333953 shutdown_connections 2023-07-20T18:53:37.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2ca80389e0 0x7f2ca803aea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:37.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc00ff110 0x7f2cc01987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:37.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cc0100430 0x7f2cc019fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:37.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 --2- 172.21.15.57:0/3486333953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cc00ffa70 0x7f2cc0198d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:37.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3486333953 >> 172.21.15.57:0/3486333953 conn(0x7f2cc00facf0 msgr2=0x7f2cc00fc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:53:37.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3486333953 shutdown_connections 2023-07-20T18:53:37.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:53:37.127+0000 7f2cc4cb2700 1 -- 172.21.15.57:0/3486333953 wait complete. 2023-07-20T18:53:37.672 DEBUG:teuthology.orchestra.run.smithi057:osd.0> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.0.service 2023-07-20T18:53:37.676 INFO:tasks.cephadm:Deploying osd.1 on smithi088 with /dev/vg_nvme/lv_4... 2023-07-20T18:53:37.677 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T18:53:37.758 INFO:journalctl@ceph.osd.0.smithi057.stdout:-- Logs begin at Thu 2023-07-20 18:30:45 UTC. -- 2023-07-20T18:53:38.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:37 smithi088 ceph-mon[128823]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:38.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:37 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:38.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:37 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:38.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:37 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:38.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:37 smithi088 ceph-mon[128823]: from='osd.0 [v2:172.21.15.57:6802/3023441607,v1:172.21.15.57:6803/3023441607]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T18:53:38.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:37 smithi088 ceph-mon[128823]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T18:53:38.226 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:37 smithi057 ceph-mon[126340]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:38.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:38.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:38.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:38.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:37 smithi057 ceph-mon[126340]: from='osd.0 [v2:172.21.15.57:6802/3023441607,v1:172.21.15.57:6803/3023441607]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T18:53:38.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:37 smithi057 ceph-mon[126340]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T18:53:38.236 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T18:53:38.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:37 smithi192 ceph-mon[129200]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:38.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:37 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:38.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:37 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:38.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:37 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:38.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:37 smithi192 ceph-mon[129200]: from='osd.0 [v2:172.21.15.57:6802/3023441607,v1:172.21.15.57:6803/3023441607]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T18:53:38.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:37 smithi192 ceph-mon[129200]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T18:53:39.281 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:39 smithi088 ceph-mon[128823]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T18:53:39.282 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:39 smithi088 ceph-mon[128823]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T18:53:39.282 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:39 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:39.282 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:39 smithi088 ceph-mon[128823]: from='osd.0 [v2:172.21.15.57:6802/3023441607,v1:172.21.15.57:6803/3023441607]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:53:39.282 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:39 smithi088 ceph-mon[128823]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:53:39.282 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:39 smithi088 ceph-mon[128823]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:39.477 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:53:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-0[139331]: 2023-07-20T18:53:39.145+0000 7fce2a636700 -1 osd.0 0 waiting for initial osdmap 2023-07-20T18:53:39.477 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:53:39 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-0[139331]: 2023-07-20T18:53:39.149+0000 7fce24428700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:53:39.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:39 smithi057 ceph-mon[126340]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T18:53:39.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:39 smithi057 ceph-mon[126340]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T18:53:39.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:39 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:39.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:39 smithi057 ceph-mon[126340]: from='osd.0 [v2:172.21.15.57:6802/3023441607,v1:172.21.15.57:6803/3023441607]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:53:39.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:39 smithi057 ceph-mon[126340]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:53:39.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:39 smithi057 ceph-mon[126340]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:39.581 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:39 smithi192 ceph-mon[129200]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T18:53:39.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:39 smithi192 ceph-mon[129200]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T18:53:39.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:39 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:39.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:39 smithi192 ceph-mon[129200]: from='osd.0 [v2:172.21.15.57:6802/3023441607,v1:172.21.15.57:6803/3023441607]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:53:39.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:39 smithi192 ceph-mon[129200]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:53:39.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:39 smithi192 ceph-mon[129200]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:39.705 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T18:53:39.706 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T18:53:39.707 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-20T18:53:39.707 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-20T18:53:39.707 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.010916 s, 961 MB/s 2023-07-20T18:53:39.707 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-20T18:53:40.076 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_4 2023-07-20T18:53:40.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:40 smithi088 ceph-mon[128823]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:53:40.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:40 smithi088 ceph-mon[128823]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T18:53:40.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:40 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:40.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:40 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:40.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:40 smithi057 ceph-mon[126340]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:53:40.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:40 smithi057 ceph-mon[126340]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T18:53:40.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:40 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:40.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:40 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:40.581 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:40 smithi192 ceph-mon[129200]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:53:40.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:40 smithi192 ceph-mon[129200]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T18:53:40.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:40 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:40.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:40 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:40.627 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T18:53:41.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:53:41.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:53:41.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:41.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: osd.0 [v2:172.21.15.57:6802/3023441607,v1:172.21.15.57:6803/3023441607] boot 2023-07-20T18:53:41.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T18:53:41.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:41.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi057 2023-07-20T18:53:41.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:53:41.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi057 to 17272M 2023-07-20T18:53:41.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:41.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:41.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:41 smithi057 ceph-mon[126340]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:41.581 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:53:41.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:53:41.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:41.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: osd.0 [v2:172.21.15.57:6802/3023441607,v1:172.21.15.57:6803/3023441607] boot 2023-07-20T18:53:41.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T18:53:41.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:41.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi057 2023-07-20T18:53:41.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.583 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.583 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:53:41.583 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi057 to 17272M 2023-07-20T18:53:41.583 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.583 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:41.583 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:41.583 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.583 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:41 smithi192 ceph-mon[129200]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:41.610 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:53:41.610 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:53:41.610 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:41.610 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: osd.0 [v2:172.21.15.57:6802/3023441607,v1:172.21.15.57:6803/3023441607] boot 2023-07-20T18:53:41.611 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T18:53:41.611 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T18:53:41.611 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi057 2023-07-20T18:53:41.611 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.611 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.611 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:53:41.611 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi057 to 17272M 2023-07-20T18:53:41.611 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.612 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:41.612 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:41.612 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:41.612 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:41 smithi088 ceph-mon[128823]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T18:53:41.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.741+0000 7efe29a89700 1 -- 172.21.15.88:0/806450654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe24101a30 msgr2=0x7efe24101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:41.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.741+0000 7efe29a89700 1 --2- 172.21.15.88:0/806450654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe24101a30 0x7efe24101e90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efe140099c0 tx=0x7efe14009cd0 comp rx=0 tx=0).stop 2023-07-20T18:53:41.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.741+0000 7efe29a89700 1 -- 172.21.15.88:0/806450654 shutdown_connections 2023-07-20T18:53:41.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.741+0000 7efe29a89700 1 --2- 172.21.15.88:0/806450654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe241023d0 0x7efe24108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.741+0000 7efe29a89700 1 --2- 172.21.15.88:0/806450654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe24100830 0x7efe24100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.741+0000 7efe29a89700 1 --2- 172.21.15.88:0/806450654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe24101a30 0x7efe24101e90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.741+0000 7efe29a89700 1 -- 172.21.15.88:0/806450654 >> 172.21.15.88:0/806450654 conn(0x7efe240fbdf0 msgr2=0x7efe240fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:53:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.741+0000 7efe29a89700 1 -- 172.21.15.88:0/806450654 shutdown_connections 2023-07-20T18:53:41.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.741+0000 7efe29a89700 1 -- 172.21.15.88:0/806450654 wait complete. 2023-07-20T18:53:41.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe29a89700 1 Processor -- start 2023-07-20T18:53:41.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe29a89700 1 -- start start 2023-07-20T18:53:41.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe29a89700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe24100830 0x7efe24196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:41.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe29a89700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe24101a30 0x7efe24196ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:41.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe29a89700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe241023d0 0x7efe2419db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:41.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe29a89700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2419e120 con 0x7efe241023d0 2023-07-20T18:53:41.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe29a89700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2419e260 con 0x7efe24100830 2023-07-20T18:53:41.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe29a89700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2419e3f0 con 0x7efe24101a30 2023-07-20T18:53:41.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe237fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe24100830 0x7efe24196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:41.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe237fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe24100830 0x7efe24196590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:45926/0 (socket says 172.21.15.88:45926) 2023-07-20T18:53:41.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe237fe700 1 -- 172.21.15.88:0/1586678222 learned_addr learned my addr 172.21.15.88:0/1586678222 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-20T18:53:41.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe22ffd700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe24101a30 0x7efe24196ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:41.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.742+0000 7efe23fff700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe241023d0 0x7efe2419db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:41.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe237fe700 1 -- 172.21.15.88:0/1586678222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe24101a30 msgr2=0x7efe24196ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:41.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe237fe700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe24101a30 0x7efe24196ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:41.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe237fe700 1 -- 172.21.15.88:0/1586678222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe241023d0 msgr2=0x7efe2419db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:41.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe237fe700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe241023d0 0x7efe2419db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:41.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe237fe700 1 -- 172.21.15.88:0/1586678222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe140095f0 con 0x7efe24100830 2023-07-20T18:53:41.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe22ffd700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe24101a30 0x7efe24196ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T18:53:41.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe237fe700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe24100830 0x7efe24196590 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efe1800b500 tx=0x7efe1800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:53:41.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe20ff9700 1 -- 172.21.15.88:0/1586678222 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18011830 con 0x7efe24100830 2023-07-20T18:53:41.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe29a89700 1 -- 172.21.15.88:0/1586678222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2419e6d0 con 0x7efe24100830 2023-07-20T18:53:41.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe20ff9700 1 -- 172.21.15.88:0/1586678222 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7efe18011e70 con 0x7efe24100830 2023-07-20T18:53:41.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe20ff9700 1 -- 172.21.15.88:0/1586678222 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1800f2e0 con 0x7efe24100830 2023-07-20T18:53:41.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.743+0000 7efe29a89700 1 -- 172.21.15.88:0/1586678222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2419ebd0 con 0x7efe24100830 2023-07-20T18:53:41.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.744+0000 7efe29a89700 1 -- 172.21.15.88:0/1586678222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe24107ce0 con 0x7efe24100830 2023-07-20T18:53:41.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.745+0000 7efe20ff9700 1 -- 172.21.15.88:0/1586678222 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7efe180105f0 con 0x7efe24100830 2023-07-20T18:53:41.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.745+0000 7efe20ff9700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efe0c0a1140 0x7efe0c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:53:41.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.746+0000 7efe20ff9700 1 -- 172.21.15.88:0/1586678222 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7efe180ca9c0 con 0x7efe24100830 2023-07-20T18:53:41.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.746+0000 7efe22ffd700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efe0c0a1140 0x7efe0c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:53:41.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.746+0000 7efe22ffd700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efe0c0a1140 0x7efe0c0a3600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efe1400b580 tx=0x7efe14009f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:53:41.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.750+0000 7efe20ff9700 1 -- 172.21.15.88:0/1586678222 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efe18095ae0 con 0x7efe24100830 2023-07-20T18:53:41.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:41.944+0000 7efe29a89700 1 -- 172.21.15.88:0/1586678222 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7efe24060bb0 con 0x7efe0c0a1140 2023-07-20T18:53:42.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:42 smithi057 ceph-mon[126340]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T18:53:42.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:42 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:53:42.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:42 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:53:42.477 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:42 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:42.581 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:42 smithi192 ceph-mon[129200]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T18:53:42.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:42 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:53:42.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:42 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:53:42.582 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:42 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:42.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:42 smithi088 ceph-mon[128823]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T18:53:42.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:42 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:53:42.624 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:42 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:53:42.625 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:42 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:44.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:43 smithi192 ceph-mon[129200]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:53:44.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:43 smithi192 ceph-mon[129200]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:44.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:43 smithi088 ceph-mon[128823]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:53:44.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:43 smithi088 ceph-mon[128823]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:44.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:43 smithi057 ceph-mon[126340]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:53:44.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:43 smithi057 ceph-mon[126340]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:45.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:44 smithi192 ceph-mon[129200]: from='client.? 172.21.15.88:0/515061214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adcdb236-334e-4478-98a8-7d0628492082"}]: dispatch 2023-07-20T18:53:45.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:44 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adcdb236-334e-4478-98a8-7d0628492082"}]: dispatch 2023-07-20T18:53:45.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:44 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "adcdb236-334e-4478-98a8-7d0628492082"}]': finished 2023-07-20T18:53:45.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:44 smithi192 ceph-mon[129200]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T18:53:45.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:44 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:53:45.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:44 smithi088 ceph-mon[128823]: from='client.? 172.21.15.88:0/515061214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adcdb236-334e-4478-98a8-7d0628492082"}]: dispatch 2023-07-20T18:53:45.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:44 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adcdb236-334e-4478-98a8-7d0628492082"}]: dispatch 2023-07-20T18:53:45.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:44 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "adcdb236-334e-4478-98a8-7d0628492082"}]': finished 2023-07-20T18:53:45.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:44 smithi088 ceph-mon[128823]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T18:53:45.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:44 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:53:45.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:44 smithi057 ceph-mon[126340]: from='client.? 172.21.15.88:0/515061214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adcdb236-334e-4478-98a8-7d0628492082"}]: dispatch 2023-07-20T18:53:45.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:44 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adcdb236-334e-4478-98a8-7d0628492082"}]: dispatch 2023-07-20T18:53:45.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:44 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "adcdb236-334e-4478-98a8-7d0628492082"}]': finished 2023-07-20T18:53:45.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:44 smithi057 ceph-mon[126340]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T18:53:45.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:44 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:53:46.081 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:45 smithi192 ceph-mon[129200]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:46.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:45 smithi192 ceph-mon[129200]: from='client.? 172.21.15.88:0/927177632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:53:46.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:45 smithi088 ceph-mon[128823]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:46.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:45 smithi088 ceph-mon[128823]: from='client.? 172.21.15.88:0/927177632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:53:46.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:45 smithi057 ceph-mon[126340]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:46.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:45 smithi057 ceph-mon[126340]: from='client.? 172.21.15.88:0/927177632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:53:48.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:47 smithi088 ceph-mon[128823]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:48.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:47 smithi057 ceph-mon[126340]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:48.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:47 smithi192 ceph-mon[129200]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:50.118 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:49 smithi088 ceph-mon[128823]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:50.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:49 smithi057 ceph-mon[126340]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:50.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:49 smithi192 ceph-mon[129200]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:52.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:51 smithi088 ceph-mon[128823]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:52.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T18:53:52.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:52.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:51 smithi088 ceph-mon[128823]: Deploying daemon osd.1 on smithi088 2023-07-20T18:53:52.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:51 smithi057 ceph-mon[126340]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:52.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T18:53:52.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:52.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:51 smithi057 ceph-mon[126340]: Deploying daemon osd.1 on smithi088 2023-07-20T18:53:52.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:51 smithi192 ceph-mon[129200]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:52.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T18:53:52.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:52.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:51 smithi192 ceph-mon[129200]: Deploying daemon osd.1 on smithi088 2023-07-20T18:53:54.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:53 smithi088 ceph-mon[128823]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:54.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:53 smithi057 ceph-mon[126340]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:54.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:53 smithi192 ceph-mon[129200]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:56.023 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:55 smithi088 ceph-mon[128823]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:56.023 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:55 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:56.024 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:55 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:56.024 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:55 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:56.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:55 smithi057 ceph-mon[126340]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:56.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:55 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:56.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:55 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:56.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:55 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:56.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:55 smithi192 ceph-mon[129200]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:56.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:55 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:56.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:55 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:56.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:55 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.911+0000 7efe20ff9700 1 -- 172.21.15.88:0/1586678222 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efe24060bb0 con 0x7efe0c0a1140 2023-07-20T18:53:57.954 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 1 on host 'smithi088' 2023-07-20T18:53:57.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.913+0000 7efe29a89700 1 -- 172.21.15.88:0/1586678222 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efe0c0a1140 msgr2=0x7efe0c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:57.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.913+0000 7efe29a89700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efe0c0a1140 0x7efe0c0a3600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efe1400b580 tx=0x7efe14009f10 comp rx=0 tx=0).stop 2023-07-20T18:53:57.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.913+0000 7efe29a89700 1 -- 172.21.15.88:0/1586678222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe24100830 msgr2=0x7efe24196590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:53:57.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.913+0000 7efe29a89700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe24100830 0x7efe24196590 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efe1800b500 tx=0x7efe1800b8c0 comp rx=0 tx=0).stop 2023-07-20T18:53:57.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.913+0000 7efe29a89700 1 -- 172.21.15.88:0/1586678222 shutdown_connections 2023-07-20T18:53:57.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.913+0000 7efe29a89700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efe0c0a1140 0x7efe0c0a3600 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:57.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.913+0000 7efe29a89700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe241023d0 0x7efe2419db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:57.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.913+0000 7efe29a89700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe24100830 0x7efe24196590 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:57.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.913+0000 7efe29a89700 1 --2- 172.21.15.88:0/1586678222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe24101a30 0x7efe24196ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:53:57.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.913+0000 7efe29a89700 1 -- 172.21.15.88:0/1586678222 >> 172.21.15.88:0/1586678222 conn(0x7efe240fbdf0 msgr2=0x7efe24104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:53:57.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.914+0000 7efe29a89700 1 -- 172.21.15.88:0/1586678222 shutdown_connections 2023-07-20T18:53:57.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:53:57.914+0000 7efe29a89700 1 -- 172.21.15.88:0/1586678222 wait complete. 2023-07-20T18:53:58.109 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:57 smithi088 ceph-mon[128823]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:58.109 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:58.109 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:58.109 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:58.109 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:58.109 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:58.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:57 smithi057 ceph-mon[126340]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:58.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:58.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:58.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:58.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:58.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:58.316 DEBUG:teuthology.orchestra.run.smithi088:osd.1> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.1.service 2023-07-20T18:53:58.318 INFO:tasks.cephadm:Deploying osd.2 on smithi192 with /dev/vg_nvme/lv_4... 2023-07-20T18:53:58.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T18:53:58.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:57 smithi192 ceph-mon[129200]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:53:58.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:58.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:58.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:53:58.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:53:58.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:58.346 INFO:journalctl@ceph.osd.1.smithi088.stdout:-- Logs begin at Thu 2023-07-20 18:25:07 UTC. -- 2023-07-20T18:53:58.880 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:53:59.117 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:59.117 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:59.117 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:59.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:59.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:59.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:59.257 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:59.258 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:53:59.258 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:53:59.375 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:53:59 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-1[133946]: 2023-07-20T18:53:59.213+0000 7fa39df70640 -1 osd.1 0 log_to_monitors true 2023-07-20T18:54:00.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:59 smithi057 ceph-mon[126340]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:54:00.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:59 smithi057 ceph-mon[126340]: from='osd.1 [v2:172.21.15.88:6800/1809528555,v1:172.21.15.88:6801/1809528555]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T18:54:00.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:53:59 smithi057 ceph-mon[126340]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T18:54:00.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:59 smithi192 ceph-mon[129200]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:54:00.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:59 smithi192 ceph-mon[129200]: from='osd.1 [v2:172.21.15.88:6800/1809528555,v1:172.21.15.88:6801/1809528555]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T18:54:00.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:53:59 smithi192 ceph-mon[129200]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T18:54:00.357 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T18:54:00.357 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T18:54:00.357 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-07-20T18:54:00.357 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-07-20T18:54:00.358 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110027 s, 953 MB/s 2023-07-20T18:54:00.358 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-07-20T18:54:00.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:59 smithi088 ceph-mon[128823]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:54:00.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:59 smithi088 ceph-mon[128823]: from='osd.1 [v2:172.21.15.88:6800/1809528555,v1:172.21.15.88:6801/1809528555]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T18:54:00.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:53:59 smithi088 ceph-mon[128823]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T18:54:00.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2023-07-20T18:54:01.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T18:54:01.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T18:54:01.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='osd.1 [v2:172.21.15.88:6800/1809528555,v1:172.21.15.88:6801/1809528555]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:54:01.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:01.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:54:01.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:01.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:01.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:01.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T18:54:01.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T18:54:01.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='osd.1 [v2:172.21.15.88:6800/1809528555,v1:172.21.15.88:6801/1809528555]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:54:01.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:01.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:54:01.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:01.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:01.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:01.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.332 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:54:01.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T18:54:01.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T18:54:01.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='osd.1 [v2:172.21.15.88:6800/1809528555,v1:172.21.15.88:6801/1809528555]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:54:01.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:01.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:54:01.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:01.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:01.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:01.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:01.377 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-1[133946]: 2023-07-20T18:54:00.913+0000 7fa3935e8700 -1 osd.1 0 waiting for initial osdmap 2023-07-20T18:54:01.378 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:54:00 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-1[133946]: 2023-07-20T18:54:00.917+0000 7fa38ebdd700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:54:02.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:01 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi088 2023-07-20T18:54:02.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:01 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi088 to 17168M 2023-07-20T18:54:02.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:01 smithi192 ceph-mon[129200]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:54:02.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:01 smithi192 ceph-mon[129200]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:54:02.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:01 smithi192 ceph-mon[129200]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T18:54:02.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:02.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:02.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:01 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi088 2023-07-20T18:54:02.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:01 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi088 to 17168M 2023-07-20T18:54:02.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:01 smithi057 ceph-mon[126340]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:54:02.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:01 smithi057 ceph-mon[126340]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:54:02.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:01 smithi057 ceph-mon[126340]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T18:54:02.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:02.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:02.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:01 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi088 2023-07-20T18:54:02.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:01 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi088 to 17168M 2023-07-20T18:54:02.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:01 smithi088 ceph-mon[128823]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:54:02.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:01 smithi088 ceph-mon[128823]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:54:02.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:01 smithi088 ceph-mon[128823]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T18:54:02.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:02.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:02.529 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.483+0000 7fa9d59af700 1 -- 172.21.15.192:0/523592313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9d0100a30 msgr2=0x7fa9d0107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:02.530 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.483+0000 7fa9d59af700 1 --2- 172.21.15.192:0/523592313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9d0100a30 0x7fa9d0107350 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800b0a0 tx=0x7fa9c800b3b0 comp rx=0 tx=0).stop 2023-07-20T18:54:02.530 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.483+0000 7fa9d59af700 1 -- 172.21.15.192:0/523592313 shutdown_connections 2023-07-20T18:54:02.530 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.483+0000 7fa9d59af700 1 --2- 172.21.15.192:0/523592313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9d0100a30 0x7fa9d0107350 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:02.531 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.483+0000 7fa9d59af700 1 --2- 172.21.15.192:0/523592313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9d00fee80 0x7fa9d00ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:02.531 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.483+0000 7fa9d59af700 1 --2- 172.21.15.192:0/523592313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9d0100070 0x7fa9d01004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:02.531 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.483+0000 7fa9d59af700 1 -- 172.21.15.192:0/523592313 >> 172.21.15.192:0/523592313 conn(0x7fa9d00fa450 msgr2=0x7fa9d00fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:54:02.532 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.483+0000 7fa9d59af700 1 -- 172.21.15.192:0/523592313 shutdown_connections 2023-07-20T18:54:02.532 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.484+0000 7fa9d59af700 1 -- 172.21.15.192:0/523592313 wait complete. 2023-07-20T18:54:02.532 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.484+0000 7fa9d59af700 1 Processor -- start 2023-07-20T18:54:02.532 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.484+0000 7fa9d59af700 1 -- start start 2023-07-20T18:54:02.533 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.484+0000 7fa9d59af700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9d00fee80 0x7fa9d0102110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:02.533 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.484+0000 7fa9d59af700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9d0100070 0x7fa9d019d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:02.533 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.484+0000 7fa9d59af700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9d0100a30 0x7fa9d019fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:02.533 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.484+0000 7fa9d59af700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d01a03d0 con 0x7fa9d0100a30 2023-07-20T18:54:02.533 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.484+0000 7fa9d59af700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d01a0540 con 0x7fa9d0100070 2023-07-20T18:54:02.534 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.484+0000 7fa9d59af700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d01a0700 con 0x7fa9d00fee80 2023-07-20T18:54:02.534 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9ceffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9d00fee80 0x7fa9d0102110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:02.534 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9ceffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9d00fee80 0x7fa9d0102110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:48128/0 (socket says 172.21.15.192:48128) 2023-07-20T18:54:02.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9ceffd700 1 -- 172.21.15.192:0/1304816152 learned_addr learned my addr 172.21.15.192:0/1304816152 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-07-20T18:54:02.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9cf7fe700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9d0100a30 0x7fa9d019fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:02.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9ce7fc700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9d0100070 0x7fa9d019d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:02.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9ceffd700 1 -- 172.21.15.192:0/1304816152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9d0100070 msgr2=0x7fa9d019d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:02.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9ceffd700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9d0100070 0x7fa9d019d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:02.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9ceffd700 1 -- 172.21.15.192:0/1304816152 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9d0100a30 msgr2=0x7fa9d019fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:02.536 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9ceffd700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9d0100a30 0x7fa9d019fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:02.536 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9ceffd700 1 -- 172.21.15.192:0/1304816152 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8009d00 con 0x7fa9d00fee80 2023-07-20T18:54:02.536 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9ceffd700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9d00fee80 0x7fa9d0102110 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000b500 tx=0x7fa9c000b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:54:02.536 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9d49ad700 1 -- 172.21.15.192:0/1304816152 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0011830 con 0x7fa9d00fee80 2023-07-20T18:54:02.536 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9d49ad700 1 -- 172.21.15.192:0/1304816152 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fa9c0011e70 con 0x7fa9d00fee80 2023-07-20T18:54:02.536 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9d49ad700 1 -- 172.21.15.192:0/1304816152 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c000f2e0 con 0x7fa9d00fee80 2023-07-20T18:54:02.536 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9d59af700 1 -- 172.21.15.192:0/1304816152 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d01a09e0 con 0x7fa9d00fee80 2023-07-20T18:54:02.537 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.485+0000 7fa9d59af700 1 -- 172.21.15.192:0/1304816152 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d01a0f10 con 0x7fa9d00fee80 2023-07-20T18:54:02.537 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.486+0000 7fa9d59af700 1 -- 172.21.15.192:0/1304816152 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d0106e80 con 0x7fa9d00fee80 2023-07-20T18:54:02.538 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.487+0000 7fa9d49ad700 1 -- 172.21.15.192:0/1304816152 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fa9c00105f0 con 0x7fa9d00fee80 2023-07-20T18:54:02.538 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.488+0000 7fa9d49ad700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa9b80a10c0 0x7fa9b80a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:02.538 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.488+0000 7fa9d49ad700 1 -- 172.21.15.192:0/1304816152 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fa9c00cab30 con 0x7fa9d00fee80 2023-07-20T18:54:02.538 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.488+0000 7fa9ce7fc700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa9b80a10c0 0x7fa9b80a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:02.538 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.488+0000 7fa9ce7fc700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa9b80a10c0 0x7fa9b80a3580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4006fd0 tx=0x7fa9c4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:54:02.538 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.492+0000 7fa9d49ad700 1 -- 172.21.15.192:0/1304816152 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa9c0095ad0 con 0x7fa9d00fee80 2023-07-20T18:54:02.725 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:02.682+0000 7fa9d59af700 1 -- 172.21.15.192:0/1304816152 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9d0060bb0 con 0x7fa9b80a10c0 2023-07-20T18:54:03.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:02 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:54:03.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:02 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:54:03.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:03.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:02 smithi057 ceph-mon[126340]: osd.1 [v2:172.21.15.88:6800/1809528555,v1:172.21.15.88:6801/1809528555] boot 2023-07-20T18:54:03.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:02 smithi057 ceph-mon[126340]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T18:54:03.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:03.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:54:03.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:54:03.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:03.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:02 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:54:03.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:02 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:54:03.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:03.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:02 smithi192 ceph-mon[129200]: osd.1 [v2:172.21.15.88:6800/1809528555,v1:172.21.15.88:6801/1809528555] boot 2023-07-20T18:54:03.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:02 smithi192 ceph-mon[129200]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T18:54:03.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:03.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:54:03.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:54:03.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:03.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:02 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:54:03.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:02 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:54:03.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:03.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:02 smithi088 ceph-mon[128823]: osd.1 [v2:172.21.15.88:6800/1809528555,v1:172.21.15.88:6801/1809528555] boot 2023-07-20T18:54:03.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:02 smithi088 ceph-mon[128823]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T18:54:03.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T18:54:03.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:54:03.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:54:03.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:04.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:03 smithi192 ceph-mon[129200]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:54:04.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:03 smithi192 ceph-mon[129200]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:54:04.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:03 smithi192 ceph-mon[129200]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T18:54:04.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:03 smithi057 ceph-mon[126340]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:54:04.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:03 smithi057 ceph-mon[126340]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:54:04.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:03 smithi057 ceph-mon[126340]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T18:54:04.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:03 smithi088 ceph-mon[128823]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T18:54:04.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:03 smithi088 ceph-mon[128823]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:54:04.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:03 smithi088 ceph-mon[128823]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T18:54:06.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:05 smithi057 ceph-mon[126340]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:06.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:05 smithi057 ceph-mon[126340]: from='client.? 172.21.15.192:0/1337664773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cd56229-2067-433a-b9f9-f47b1b878b66"}]: dispatch 2023-07-20T18:54:06.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:05 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cd56229-2067-433a-b9f9-f47b1b878b66"}]: dispatch 2023-07-20T18:54:06.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:05 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cd56229-2067-433a-b9f9-f47b1b878b66"}]': finished 2023-07-20T18:54:06.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:05 smithi057 ceph-mon[126340]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T18:54:06.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:05 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:06.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:05 smithi057 ceph-mon[126340]: from='client.? 172.21.15.192:0/1627124297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:54:06.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:05 smithi192 ceph-mon[129200]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:06.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:05 smithi192 ceph-mon[129200]: from='client.? 172.21.15.192:0/1337664773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cd56229-2067-433a-b9f9-f47b1b878b66"}]: dispatch 2023-07-20T18:54:06.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:05 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cd56229-2067-433a-b9f9-f47b1b878b66"}]: dispatch 2023-07-20T18:54:06.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:05 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cd56229-2067-433a-b9f9-f47b1b878b66"}]': finished 2023-07-20T18:54:06.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:05 smithi192 ceph-mon[129200]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T18:54:06.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:05 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:06.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:05 smithi192 ceph-mon[129200]: from='client.? 172.21.15.192:0/1627124297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:54:06.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:05 smithi088 ceph-mon[128823]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:06.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:05 smithi088 ceph-mon[128823]: from='client.? 172.21.15.192:0/1337664773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cd56229-2067-433a-b9f9-f47b1b878b66"}]: dispatch 2023-07-20T18:54:06.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:05 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cd56229-2067-433a-b9f9-f47b1b878b66"}]: dispatch 2023-07-20T18:54:06.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:05 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cd56229-2067-433a-b9f9-f47b1b878b66"}]': finished 2023-07-20T18:54:06.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:05 smithi088 ceph-mon[128823]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T18:54:06.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:05 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:06.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:05 smithi088 ceph-mon[128823]: from='client.? 172.21.15.192:0/1627124297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:54:08.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:07 smithi192 ceph-mon[129200]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:08.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:07 smithi088 ceph-mon[128823]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:08.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:07 smithi057 ceph-mon[126340]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:09.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:08 smithi192 ceph-mon[129200]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:09.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:08 smithi088 ceph-mon[128823]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:09.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:08 smithi057 ceph-mon[126340]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:11.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:11 smithi192 ceph-mon[129200]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:11.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T18:54:11.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:12.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:11 smithi088 ceph-mon[128823]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:12.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T18:54:12.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:12.227 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:11 smithi057 ceph-mon[126340]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:12.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T18:54:12.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:13.008 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:12 smithi192 ceph-mon[129200]: Deploying daemon osd.2 on smithi192 2023-07-20T18:54:13.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:12 smithi088 ceph-mon[128823]: Deploying daemon osd.2 on smithi192 2023-07-20T18:54:13.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:12 smithi057 ceph-mon[126340]: Deploying daemon osd.2 on smithi192 2023-07-20T18:54:13.986 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:13 smithi192 ceph-mon[129200]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:14.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:13 smithi088 ceph-mon[128823]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:14.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:13 smithi057 ceph-mon[126340]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:16.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:15 smithi192 ceph-mon[129200]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:16.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:16.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:16.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:16.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:15 smithi088 ceph-mon[128823]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:16.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:16.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:16.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:16.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:15 smithi057 ceph-mon[126340]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:16.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:16.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:16.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:18.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:17 smithi192 ceph-mon[129200]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:18.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:17 smithi088 ceph-mon[128823]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:18.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:17 smithi057 ceph-mon[126340]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:18.685 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 2 on host 'smithi192' 2023-07-20T18:54:18.686 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.642+0000 7fa9d49ad700 1 -- 172.21.15.192:0/1304816152 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa9d0060bb0 con 0x7fa9b80a10c0 2023-07-20T18:54:18.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.644+0000 7fa9d59af700 1 -- 172.21.15.192:0/1304816152 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa9b80a10c0 msgr2=0x7fa9b80a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:18.687 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.644+0000 7fa9d59af700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa9b80a10c0 0x7fa9b80a3580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4006fd0 tx=0x7fa9c4008040 comp rx=0 tx=0).stop 2023-07-20T18:54:18.688 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.645+0000 7fa9d59af700 1 -- 172.21.15.192:0/1304816152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9d00fee80 msgr2=0x7fa9d0102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:18.688 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.645+0000 7fa9d59af700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9d00fee80 0x7fa9d0102110 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000b500 tx=0x7fa9c000b8c0 comp rx=0 tx=0).stop 2023-07-20T18:54:18.688 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.645+0000 7fa9d59af700 1 -- 172.21.15.192:0/1304816152 shutdown_connections 2023-07-20T18:54:18.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.645+0000 7fa9d59af700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa9b80a10c0 0x7fa9b80a3580 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:18.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.645+0000 7fa9d59af700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9d00fee80 0x7fa9d0102110 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:18.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.645+0000 7fa9d59af700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9d0100a30 0x7fa9d019fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:18.689 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.645+0000 7fa9d59af700 1 --2- 172.21.15.192:0/1304816152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9d0100070 0x7fa9d019d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:18.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.645+0000 7fa9d59af700 1 -- 172.21.15.192:0/1304816152 >> 172.21.15.192:0/1304816152 conn(0x7fa9d00fa450 msgr2=0x7fa9d0068340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:54:18.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.645+0000 7fa9d59af700 1 -- 172.21.15.192:0/1304816152 shutdown_connections 2023-07-20T18:54:18.690 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:54:18.645+0000 7fa9d59af700 1 -- 172.21.15.192:0/1304816152 wait complete. 2023-07-20T18:54:19.110 DEBUG:teuthology.orchestra.run.smithi192:osd.2> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.2.service 2023-07-20T18:54:19.112 INFO:tasks.cephadm:Deploying osd.3 on smithi057 with /dev/vg_nvme/lv_3... 2023-07-20T18:54:19.113 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T18:54:19.139 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.140 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.140 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:19.140 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:19.140 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.140 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.140 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.141 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:19.141 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:18 smithi192 ceph-mon[129200]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:19.146 INFO:journalctl@ceph.osd.2.smithi192.stdout:-- Logs begin at Thu 2023-07-20 18:24:33 UTC. -- 2023-07-20T18:54:19.375 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:19.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:19.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.376 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:19.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:18 smithi088 ceph-mon[128823]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:19.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:19.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:19.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.480 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.480 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:19.480 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:19.480 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:18 smithi057 ceph-mon[126340]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:19.853 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:54:19.879 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:19 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-2[135031]: 2023-07-20T18:54:19.732+0000 7f166e358640 -1 osd.2 0 log_to_monitors true 2023-07-20T18:54:20.728 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:20 smithi057 ceph-mon[126340]: from='osd.2 [v2:172.21.15.192:6800/3891687382,v1:172.21.15.192:6801/3891687382]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T18:54:20.728 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:20 smithi057 ceph-mon[126340]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T18:54:20.948 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:20 smithi192 ceph-mon[129200]: from='osd.2 [v2:172.21.15.192:6800/3891687382,v1:172.21.15.192:6801/3891687382]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T18:54:20.949 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:20 smithi192 ceph-mon[129200]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T18:54:21.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:20 smithi088 ceph-mon[128823]: from='osd.2 [v2:172.21.15.192:6800/3891687382,v1:172.21.15.192:6801/3891687382]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T18:54:21.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:20 smithi088 ceph-mon[128823]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T18:54:21.864 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T18:54:21.864 INFO:teuthology.orchestra.run.smithi057.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T18:54:21.865 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10+0 records in 2023-07-20T18:54:21.865 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2023-07-20T18:54:21.865 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0301548 s, 348 MB/s 2023-07-20T18:54:21.865 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping successful for: 2023-07-20T18:54:21.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T18:54:21.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T18:54:21.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:21.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:21.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='osd.2 [v2:172.21.15.192:6800/3891687382,v1:172.21.15.192:6801/3891687382]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:54:21.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:54:21.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:21.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:21.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:21.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:21.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:21.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:21.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:22.083 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-2[135031]: 2023-07-20T18:54:21.650+0000 7f16639d0700 -1 osd.2 0 waiting for initial osdmap 2023-07-20T18:54:22.083 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-2[135031]: 2023-07-20T18:54:21.656+0000 7f165bfbf700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:54:22.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T18:54:22.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T18:54:22.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:22.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:22.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='osd.2 [v2:172.21.15.192:6800/3891687382,v1:172.21.15.192:6801/3891687382]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:54:22.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:54:22.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:22.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:22.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:22.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:22.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:22.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:22.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:22.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T18:54:22.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T18:54:22.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:22.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:22.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='osd.2 [v2:172.21.15.192:6800/3891687382,v1:172.21.15.192:6801/3891687382]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:54:22.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:54:22.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:22.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:22.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:22.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:22.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:22.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:22.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:22.435 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi057:vg_nvme/lv_3 2023-07-20T18:54:22.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:22 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi192 2023-07-20T18:54:22.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:22 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi192 to 17168M 2023-07-20T18:54:22.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:22 smithi057 ceph-mon[126340]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:54:22.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:22 smithi057 ceph-mon[126340]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T18:54:22.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:22 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:22.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:22 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:22.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:22 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:23.082 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:22 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi192 2023-07-20T18:54:23.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:22 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi192 to 17168M 2023-07-20T18:54:23.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:22 smithi192 ceph-mon[129200]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:54:23.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:22 smithi192 ceph-mon[129200]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T18:54:23.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:22 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:23.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:22 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:23.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:22 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:23.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:22 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi192 2023-07-20T18:54:23.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:22 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi192 to 17168M 2023-07-20T18:54:23.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:22 smithi088 ceph-mon[128823]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:54:23.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:22 smithi088 ceph-mon[128823]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T18:54:23.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:22 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:23.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:22 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:23.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:22 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:23.175 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:54:23.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:23 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:54:23.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:23 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:54:23.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:23 smithi057 ceph-mon[126340]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:23.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:23 smithi057 ceph-mon[126340]: osd.2 [v2:172.21.15.192:6800/3891687382,v1:172.21.15.192:6801/3891687382] boot 2023-07-20T18:54:23.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:23 smithi057 ceph-mon[126340]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T18:54:23.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:23.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T18:54:24.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:23 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:54:24.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:23 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:54:24.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:23 smithi192 ceph-mon[129200]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:24.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:23 smithi192 ceph-mon[129200]: osd.2 [v2:172.21.15.192:6800/3891687382,v1:172.21.15.192:6801/3891687382] boot 2023-07-20T18:54:24.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:23 smithi192 ceph-mon[129200]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T18:54:24.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:24.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T18:54:24.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:23 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:54:24.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:23 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:54:24.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:23 smithi088 ceph-mon[128823]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T18:54:24.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:23 smithi088 ceph-mon[128823]: osd.2 [v2:172.21.15.192:6800/3891687382,v1:172.21.15.192:6801/3891687382] boot 2023-07-20T18:54:24.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:23 smithi088 ceph-mon[128823]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T18:54:24.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T18:54:24.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T18:54:24.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.688+0000 7f0296a46700 1 -- 172.21.15.57:0/1043000919 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0290100800 msgr2=0x7f0290100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:24.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.688+0000 7f0296a46700 1 --2- 172.21.15.57:0/1043000919 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0290100800 0x7f0290100c20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f02800099c0 tx=0x7f0280009cd0 comp rx=0 tx=0).stop 2023-07-20T18:54:24.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.688+0000 7f0296a46700 1 -- 172.21.15.57:0/1043000919 shutdown_connections 2023-07-20T18:54:24.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.688+0000 7f0296a46700 1 --2- 172.21.15.57:0/1043000919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02901023a0 0x7f0290108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:24.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.688+0000 7f0296a46700 1 --2- 172.21.15.57:0/1043000919 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0290100800 0x7f0290100c20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:24.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.688+0000 7f0296a46700 1 --2- 172.21.15.57:0/1043000919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0290101a00 0x7f0290101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:24.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.688+0000 7f0296a46700 1 -- 172.21.15.57:0/1043000919 >> 172.21.15.57:0/1043000919 conn(0x7f02900fbd60 msgr2=0x7f02900fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:54:24.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.688+0000 7f0296a46700 1 -- 172.21.15.57:0/1043000919 shutdown_connections 2023-07-20T18:54:24.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.688+0000 7f0296a46700 1 -- 172.21.15.57:0/1043000919 wait complete. 2023-07-20T18:54:24.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.689+0000 7f0296a46700 1 Processor -- start 2023-07-20T18:54:24.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.689+0000 7f0296a46700 1 -- start start 2023-07-20T18:54:24.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.689+0000 7f0296a46700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0290100800 0x7f0290198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:24.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.689+0000 7f0296a46700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0290101a00 0x7f0290198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:24.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.689+0000 7f0296a46700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02901023a0 0x7f029019fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:24.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.689+0000 7f0296a46700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02901a0320 con 0x7f0290101a00 2023-07-20T18:54:24.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.689+0000 7f0296a46700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02901a0460 con 0x7f02901023a0 2023-07-20T18:54:24.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.689+0000 7f0296a46700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02901a05f0 con 0x7f0290100800 2023-07-20T18:54:24.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f028f7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0290101a00 0x7f0290198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:24.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f028f7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0290101a00 0x7f0290198cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37250/0 (socket says 172.21.15.57:37250) 2023-07-20T18:54:24.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f028f7fe700 1 -- 172.21.15.57:0/1873113081 learned_addr learned my addr 172.21.15.57:0/1873113081 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:54:24.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f028ffff700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0290100800 0x7f0290198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:24.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f0294fe3700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02901023a0 0x7f029019fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:24.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f028f7fe700 1 -- 172.21.15.57:0/1873113081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0290100800 msgr2=0x7f0290198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:24.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f028f7fe700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0290100800 0x7f0290198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:24.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f028f7fe700 1 -- 172.21.15.57:0/1873113081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02901023a0 msgr2=0x7f029019fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:24.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f028f7fe700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02901023a0 0x7f029019fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:24.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f028f7fe700 1 -- 172.21.15.57:0/1873113081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02800095f0 con 0x7f0290101a00 2023-07-20T18:54:24.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.690+0000 7f028f7fe700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0290101a00 0x7f0290198cd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f028400e9d0 tx=0x7f028400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:54:24.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.691+0000 7f028d7fa700 1 -- 172.21.15.57:0/1873113081 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284009790 con 0x7f0290101a00 2023-07-20T18:54:24.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.691+0000 7f028d7fa700 1 -- 172.21.15.57:0/1873113081 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f028400ee70 con 0x7f0290101a00 2023-07-20T18:54:24.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.691+0000 7f0296a46700 1 -- 172.21.15.57:0/1873113081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02901a08d0 con 0x7f0290101a00 2023-07-20T18:54:24.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.691+0000 7f028d7fa700 1 -- 172.21.15.57:0/1873113081 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284018680 con 0x7f0290101a00 2023-07-20T18:54:24.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.691+0000 7f0296a46700 1 -- 172.21.15.57:0/1873113081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02901a0e20 con 0x7f0290101a00 2023-07-20T18:54:24.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.692+0000 7f0296a46700 1 -- 172.21.15.57:0/1873113081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029010c140 con 0x7f0290101a00 2023-07-20T18:54:24.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.694+0000 7f028d7fa700 1 -- 172.21.15.57:0/1873113081 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f028400cbd0 con 0x7f0290101a00 2023-07-20T18:54:24.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.695+0000 7f028d7fa700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f02780a1140 0x7f02780a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:24.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.695+0000 7f028d7fa700 1 -- 172.21.15.57:0/1873113081 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f0284014070 con 0x7f0290101a00 2023-07-20T18:54:24.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.696+0000 7f028ffff700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f02780a1140 0x7f02780a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:24.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.696+0000 7f028ffff700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f02780a1140 0x7f02780a3600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0280009fd0 tx=0x7f0280005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:54:24.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.700+0000 7f028d7fa700 1 -- 172.21.15.57:0/1873113081 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0284051780 con 0x7f0290101a00 2023-07-20T18:54:24.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:24.914+0000 7f0296a46700 1 -- 172.21.15.57:0/1873113081 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0290108300 con 0x7f02780a1140 2023-07-20T18:54:24.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:24 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T18:54:24.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:24 smithi057 ceph-mon[126340]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T18:54:24.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:24 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T18:54:25.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:24 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T18:54:25.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:24 smithi192 ceph-mon[129200]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T18:54:25.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:24 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T18:54:25.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:24 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T18:54:25.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:24 smithi088 ceph-mon[128823]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T18:54:25.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:24 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T18:54:25.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:25 smithi057 ceph-mon[126340]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:25.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T18:54:25.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:25 smithi057 ceph-mon[126340]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T18:54:25.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:25 smithi057 ceph-mon[126340]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:54:25.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:54:25.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:54:25.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:25.979 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:54:25 smithi057 sudo[142360]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T18:54:25.980 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:54:25 smithi057 sudo[142360]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:25.980 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:54:25 smithi057 sudo[142360]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:25.980 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:54:25 smithi057 sudo[142360]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:26.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:25 smithi192 ceph-mon[129200]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:26.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T18:54:26.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:25 smithi192 ceph-mon[129200]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T18:54:26.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:25 smithi192 ceph-mon[129200]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:54:26.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:54:26.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:54:26.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:26.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:25 smithi088 ceph-mon[128823]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:26.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T18:54:26.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:25 smithi088 ceph-mon[128823]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T18:54:26.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:25 smithi088 ceph-mon[128823]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:54:26.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:54:26.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:54:26.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:26.728 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:54:26 smithi057 sudo[142410]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T18:54:26.728 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:54:26 smithi057 sudo[142410]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:26.728 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:54:26 smithi057 sudo[142410]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:26.729 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:54:26 smithi057 sudo[142410]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:27.041 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:26 smithi088 ceph-mon[128823]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T18:54:27.041 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:26 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:27.042 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:54:26 smithi088 sudo[136146]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T18:54:27.042 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:54:26 smithi088 sudo[136146]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:27.042 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:54:26 smithi088 sudo[136146]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:27.042 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:54:26 smithi088 sudo[136146]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:27.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:26 smithi192 ceph-mon[129200]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T18:54:27.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:26 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:27.148 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:26 smithi057 ceph-mon[126340]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T18:54:27.148 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:26 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:27.375 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:54:26 smithi088 sudo[136149]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T18:54:27.376 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:54:26 smithi088 sudo[136149]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:27.376 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:54:26 smithi088 sudo[136149]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:27.376 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:54:27 smithi088 sudo[136149]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:27.561 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:27 smithi192 sudo[137282]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T18:54:27.562 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:27 smithi192 sudo[137282]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:27.562 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:27 smithi192 sudo[137282]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:27.562 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:27 smithi192 sudo[137282]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:27.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:27.518+0000 7f028d7fa700 1 -- 172.21.15.57:0/1873113081 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0284096530 con 0x7f0290101a00 2023-07-20T18:54:27.833 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:27 smithi192 sudo[137285]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T18:54:27.833 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:27 smithi192 sudo[137285]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:27.833 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:27 smithi192 sudo[137285]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:27.833 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:54:27 smithi192 sudo[137285]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:28.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 sudo[142502]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T18:54:28.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 sudo[142502]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:28.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 sudo[142502]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:28.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 sudo[142502]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:28.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:28 smithi192 ceph-mon[129200]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:28.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:28 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:28.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:28 smithi192 ceph-mon[129200]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T18:54:28.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:28 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:28.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:28 smithi192 ceph-mon[129200]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T18:54:28.945 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:28 smithi192 sudo[137288]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T18:54:28.945 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:28 smithi192 sudo[137288]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:28.945 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:28 smithi192 sudo[137288]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:28.945 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:28 smithi192 sudo[137288]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:28.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 sudo[142506]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T18:54:28.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 sudo[142506]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:28.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 sudo[142506]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:28.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 sudo[142506]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:28.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 ceph-mon[126340]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:28.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:28.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 ceph-mon[126340]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T18:54:28.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:28.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:28 smithi057 ceph-mon[126340]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T18:54:29.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:28 smithi088 ceph-mon[128823]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:29.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:28 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:29.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:28 smithi088 ceph-mon[128823]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T18:54:29.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:28 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:29.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:28 smithi088 ceph-mon[128823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T18:54:29.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 sudo[137291]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T18:54:29.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 sudo[137291]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:29.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 sudo[137291]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:29.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 sudo[137291]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:29.675 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 sudo[136152]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T18:54:29.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 sudo[136152]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:29.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 sudo[136152]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:29.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 sudo[136152]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:29.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/4223688258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae250ab-e242-4ef8-9c3a-a4fa36102c6e"}]: dispatch 2023-07-20T18:54:29.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T18:54:29.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:54:29.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae250ab-e242-4ef8-9c3a-a4fa36102c6e"}]: dispatch 2023-07-20T18:54:29.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:29.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:54:29.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:54:29.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T18:54:29.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aae250ab-e242-4ef8-9c3a-a4fa36102c6e"}]': finished 2023-07-20T18:54:29.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T18:54:29.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:29.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:54:29.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:54:29.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:54:29.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T18:54:29.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T18:54:29.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:54:29.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:54:29.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:54:29.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:29 smithi057 ceph-mon[126340]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T18:54:30.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/4223688258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae250ab-e242-4ef8-9c3a-a4fa36102c6e"}]: dispatch 2023-07-20T18:54:30.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T18:54:30.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:54:30.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae250ab-e242-4ef8-9c3a-a4fa36102c6e"}]: dispatch 2023-07-20T18:54:30.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:30.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:54:30.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:54:30.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T18:54:30.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aae250ab-e242-4ef8-9c3a-a4fa36102c6e"}]': finished 2023-07-20T18:54:30.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T18:54:30.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:30.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:54:30.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:54:30.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:54:30.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T18:54:30.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T18:54:30.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:54:30.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:54:30.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:54:30.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:29 smithi192 ceph-mon[129200]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T18:54:30.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 sudo[136155]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T18:54:30.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 sudo[136155]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T18:54:30.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 sudo[136155]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T18:54:30.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 sudo[136155]: pam_unix(sudo:session): session closed for user root 2023-07-20T18:54:30.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/4223688258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae250ab-e242-4ef8-9c3a-a4fa36102c6e"}]: dispatch 2023-07-20T18:54:30.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T18:54:30.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:54:30.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae250ab-e242-4ef8-9c3a-a4fa36102c6e"}]: dispatch 2023-07-20T18:54:30.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:30.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:54:30.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:54:30.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T18:54:30.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aae250ab-e242-4ef8-9c3a-a4fa36102c6e"}]': finished 2023-07-20T18:54:30.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T18:54:30.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:30.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:54:30.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:54:30.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:54:30.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T18:54:30.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T18:54:30.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T18:54:30.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T18:54:30.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T18:54:30.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:29 smithi088 ceph-mon[128823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T18:54:30.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:30 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/4208023503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:54:31.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:30 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/4208023503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:54:31.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:30 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/4208023503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:54:31.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:31 smithi057 ceph-mon[126340]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:32.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:31 smithi192 ceph-mon[129200]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:32.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:31 smithi088 ceph-mon[128823]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:33.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:33 smithi057 ceph-mon[126340]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:34.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:33 smithi192 ceph-mon[129200]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:34.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:33 smithi088 ceph-mon[128823]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:35.969 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:35 smithi057 ceph-mon[126340]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-20T18:54:36.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:35 smithi192 ceph-mon[129200]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-20T18:54:36.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:35 smithi088 ceph-mon[128823]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-20T18:54:37.974 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:37 smithi057 ceph-mon[126340]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-20T18:54:37.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T18:54:37.975 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:38.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:37 smithi192 ceph-mon[129200]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-20T18:54:38.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:37 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T18:54:38.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:37 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:38.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:37 smithi088 ceph-mon[128823]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-20T18:54:38.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:37 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T18:54:38.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:37 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:38.978 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:38 smithi057 ceph-mon[126340]: Deploying daemon osd.3 on smithi057 2023-07-20T18:54:39.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:38 smithi192 ceph-mon[129200]: Deploying daemon osd.3 on smithi057 2023-07-20T18:54:39.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:38 smithi088 ceph-mon[128823]: Deploying daemon osd.3 on smithi057 2023-07-20T18:54:39.963 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:39 smithi057 ceph-mon[126340]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 920 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-20T18:54:40.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:39 smithi192 ceph-mon[129200]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 920 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-20T18:54:40.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:39 smithi088 ceph-mon[128823]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 920 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-20T18:54:41.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:41 smithi057 ceph-mon[126340]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T18:54:42.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:41 smithi192 ceph-mon[129200]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T18:54:42.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:41 smithi088 ceph-mon[128823]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T18:54:43.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:43 smithi057 ceph-mon[126340]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T18:54:43.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:43.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:43.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:44.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:43 smithi192 ceph-mon[129200]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T18:54:44.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:44.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:44.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:44.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:43 smithi088 ceph-mon[128823]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T18:54:44.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:44.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:44.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:45.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:45 smithi057 ceph-mon[126340]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T18:54:46.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:45 smithi192 ceph-mon[129200]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T18:54:46.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:45 smithi088 ceph-mon[128823]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T18:54:47.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.350+0000 7f028d7fa700 1 -- 172.21.15.57:0/1873113081 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0290108300 con 0x7f02780a1140 2023-07-20T18:54:47.395 INFO:teuthology.orchestra.run.smithi057.stdout:Created osd(s) 3 on host 'smithi057' 2023-07-20T18:54:47.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 -- 172.21.15.57:0/1873113081 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f02780a1140 msgr2=0x7f02780a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:47.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f02780a1140 0x7f02780a3600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0280009fd0 tx=0x7f0280005fb0 comp rx=0 tx=0).stop 2023-07-20T18:54:47.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 -- 172.21.15.57:0/1873113081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0290101a00 msgr2=0x7f0290198cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:47.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0290101a00 0x7f0290198cd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f028400e9d0 tx=0x7f028400ece0 comp rx=0 tx=0).stop 2023-07-20T18:54:47.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 -- 172.21.15.57:0/1873113081 shutdown_connections 2023-07-20T18:54:47.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f02780a1140 0x7f02780a3600 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:47.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0290100800 0x7f0290198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:47.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02901023a0 0x7f029019fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:47.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 --2- 172.21.15.57:0/1873113081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0290101a00 0x7f0290198cd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:47.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 -- 172.21.15.57:0/1873113081 >> 172.21.15.57:0/1873113081 conn(0x7f02900fbd60 msgr2=0x7f02900fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:54:47.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 -- 172.21.15.57:0/1873113081 shutdown_connections 2023-07-20T18:54:47.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:54:47.353+0000 7f0296a46700 1 -- 172.21.15.57:0/1873113081 wait complete. 2023-07-20T18:54:47.420 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:47 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:47.420 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:47 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:47.420 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:47 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:47.421 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:47 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:47.421 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:47 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:47.421 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:47 smithi057 ceph-mon[126340]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:47.583 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:47 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:47.584 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:47 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:47.584 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:47 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:47.584 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:47 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:47.584 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:47 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:47.584 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:47 smithi192 ceph-mon[129200]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:47.626 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:47 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:47.626 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:47 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:47.626 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:47 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:47.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:47 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:47.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:47 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:47.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:47 smithi088 ceph-mon[128823]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:47.805 DEBUG:teuthology.orchestra.run.smithi057:osd.3> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.3.service 2023-07-20T18:54:47.809 INFO:tasks.cephadm:Deploying osd.4 on smithi088 with /dev/vg_nvme/lv_3... 2023-07-20T18:54:47.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T18:54:47.882 INFO:journalctl@ceph.osd.3.smithi057.stdout:-- Logs begin at Thu 2023-07-20 18:30:45 UTC. -- 2023-07-20T18:54:48.370 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T18:54:48.601 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:48 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:48.601 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:48 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:48.601 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:48 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:48.602 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:48 smithi088 ceph-mon[128823]: from='osd.3 [v2:172.21.15.57:6810/2920717314,v1:172.21.15.57:6811/2920717314]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T18:54:48.602 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:48 smithi088 ceph-mon[128823]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T18:54:48.638 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:48 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:48.638 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:48 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:48.638 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:48 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:48.639 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:48 smithi057 ceph-mon[126340]: from='osd.3 [v2:172.21.15.57:6810/2920717314,v1:172.21.15.57:6811/2920717314]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T18:54:48.639 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:48 smithi057 ceph-mon[126340]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T18:54:48.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:48 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:48.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:48 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:48.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:48 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:54:48.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:48 smithi192 ceph-mon[129200]: from='osd.3 [v2:172.21.15.57:6810/2920717314,v1:172.21.15.57:6811/2920717314]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T18:54:48.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:48 smithi192 ceph-mon[129200]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T18:54:49.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:49 smithi057 ceph-mon[126340]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T18:54:49.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:49 smithi057 ceph-mon[126340]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T18:54:49.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:49 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:49.479 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:49 smithi057 ceph-mon[126340]: from='osd.3 [v2:172.21.15.57:6810/2920717314,v1:172.21.15.57:6811/2920717314]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:54:49.480 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:49 smithi057 ceph-mon[126340]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:54:49.480 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:49 smithi057 ceph-mon[126340]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:49.480 INFO:journalctl@ceph.osd.3.smithi057.stdout:Jul 20 18:54:49 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-3[144749]: 2023-07-20T18:54:49.321+0000 7fac22651700 -1 osd.3 0 waiting for initial osdmap 2023-07-20T18:54:49.480 INFO:journalctl@ceph.osd.3.smithi057.stdout:Jul 20 18:54:49 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-3[144749]: 2023-07-20T18:54:49.330+0000 7fac1b441700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:54:49.626 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:49 smithi088 ceph-mon[128823]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T18:54:49.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:49 smithi088 ceph-mon[128823]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T18:54:49.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:49 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:49.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:49 smithi088 ceph-mon[128823]: from='osd.3 [v2:172.21.15.57:6810/2920717314,v1:172.21.15.57:6811/2920717314]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:54:49.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:49 smithi088 ceph-mon[128823]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:54:49.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:49 smithi088 ceph-mon[128823]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:49.754 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T18:54:49.754 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T18:54:49.754 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-20T18:54:49.754 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-20T18:54:49.755 INFO:teuthology.orchestra.run.smithi088.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148927 s, 704 MB/s 2023-07-20T18:54:49.755 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-20T18:54:49.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:49 smithi192 ceph-mon[129200]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T18:54:49.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:49 smithi192 ceph-mon[129200]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T18:54:49.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:49 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:49.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:49 smithi192 ceph-mon[129200]: from='osd.3 [v2:172.21.15.57:6810/2920717314,v1:172.21.15.57:6811/2920717314]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:54:49.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:49 smithi192 ceph-mon[129200]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:54:49.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:49 smithi192 ceph-mon[129200]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T18:54:50.126 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_3 2023-07-20T18:54:50.626 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:50 smithi088 ceph-mon[128823]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:54:50.626 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:50 smithi088 ceph-mon[128823]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T18:54:50.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:50 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:50.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:50 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:50.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:50 smithi088 ceph-mon[128823]: osd.3 [v2:172.21.15.57:6810/2920717314,v1:172.21.15.57:6811/2920717314] boot 2023-07-20T18:54:50.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:50 smithi088 ceph-mon[128823]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T18:54:50.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:50 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:50.677 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T18:54:50.688 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:50 smithi057 ceph-mon[126340]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:54:50.689 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:50 smithi057 ceph-mon[126340]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T18:54:50.689 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:50.689 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:50.689 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:50 smithi057 ceph-mon[126340]: osd.3 [v2:172.21.15.57:6810/2920717314,v1:172.21.15.57:6811/2920717314] boot 2023-07-20T18:54:50.689 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:50 smithi057 ceph-mon[126340]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T18:54:50.689 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:50.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:50 smithi192 ceph-mon[129200]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:54:50.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:50 smithi192 ceph-mon[129200]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T18:54:50.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:50.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:50.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:50 smithi192 ceph-mon[129200]: osd.3 [v2:172.21.15.57:6810/2920717314,v1:172.21.15.57:6811/2920717314] boot 2023-07-20T18:54:50.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:50 smithi192 ceph-mon[129200]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T18:54:50.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T18:54:51.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:54:51.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:54:51.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:54:51.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi057 2023-07-20T18:54:51.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:51.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:51.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi057 to 8636M 2023-07-20T18:54:51.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:51.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:51.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:51 smithi088 ceph-mon[128823]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T18:54:51.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:54:51.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:54:51.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:54:51.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi057 2023-07-20T18:54:51.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:51.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:51.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi057 to 8636M 2023-07-20T18:54:51.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:51.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:51.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:51 smithi057 ceph-mon[126340]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T18:54:51.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 -- 172.21.15.88:0/289626632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4578100900 msgr2=0x7f4578100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:51.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 --2- 172.21.15.88:0/289626632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4578100900 0x7f4578100d60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f45680099c0 tx=0x7f4568009cd0 comp rx=0 tx=0).stop 2023-07-20T18:54:51.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 -- 172.21.15.88:0/289626632 shutdown_connections 2023-07-20T18:54:51.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 --2- 172.21.15.88:0/289626632 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45781012a0 0x7f4578107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:51.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 --2- 172.21.15.88:0/289626632 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45780ff700 0x7f45780ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:51.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 --2- 172.21.15.88:0/289626632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4578100900 0x7f4578100d60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:51.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 -- 172.21.15.88:0/289626632 >> 172.21.15.88:0/289626632 conn(0x7f45780fac80 msgr2=0x7f45780fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:54:51.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 -- 172.21.15.88:0/289626632 shutdown_connections 2023-07-20T18:54:51.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 -- 172.21.15.88:0/289626632 wait complete. 2023-07-20T18:54:51.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 Processor -- start 2023-07-20T18:54:51.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 -- start start 2023-07-20T18:54:51.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.691+0000 7f457f5db700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45780ff700 0x7f4578068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:51.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457f5db700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4578100900 0x7f457819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:51.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457f5db700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45781012a0 0x7f457819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:51.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457f5db700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457819e170 con 0x7f4578100900 2023-07-20T18:54:51.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457f5db700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457819e2e0 con 0x7f45781012a0 2023-07-20T18:54:51.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457f5db700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457819e4a0 con 0x7f45780ff700 2023-07-20T18:54:51.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457db78700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45781012a0 0x7f457819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:51.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457db78700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45781012a0 0x7f457819d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46008/0 (socket says 172.21.15.88:46008) 2023-07-20T18:54:51.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457db78700 1 -- 172.21.15.88:0/166240311 learned_addr learned my addr 172.21.15.88:0/166240311 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-20T18:54:51.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457d377700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45780ff700 0x7f4578068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:51.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457db78700 1 -- 172.21.15.88:0/166240311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45780ff700 msgr2=0x7f4578068d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:51.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457db78700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45780ff700 0x7f4578068d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:51.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457db78700 1 -- 172.21.15.88:0/166240311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4578100900 msgr2=0x7f457819b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:54:51.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457db78700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4578100900 0x7f457819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:54:51.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457db78700 1 -- 172.21.15.88:0/166240311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45680095f0 con 0x7f45781012a0 2023-07-20T18:54:51.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457db78700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45781012a0 0x7f457819d9d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f457000ada0 tx=0x7f457000c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:54:51.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f456e7fc700 1 -- 172.21.15.88:0/166240311 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457000f070 con 0x7f45781012a0 2023-07-20T18:54:51.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457f5db700 1 -- 172.21.15.88:0/166240311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457819e780 con 0x7f45781012a0 2023-07-20T18:54:51.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f456e7fc700 1 -- 172.21.15.88:0/166240311 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f4570007b30 con 0x7f45781012a0 2023-07-20T18:54:51.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f456e7fc700 1 -- 172.21.15.88:0/166240311 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45700053c0 con 0x7f45781012a0 2023-07-20T18:54:51.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.692+0000 7f457f5db700 1 -- 172.21.15.88:0/166240311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457819ec80 con 0x7f45781012a0 2023-07-20T18:54:51.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.693+0000 7f457f5db700 1 -- 172.21.15.88:0/166240311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4578106cd0 con 0x7f45781012a0 2023-07-20T18:54:51.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.695+0000 7f456e7fc700 1 -- 172.21.15.88:0/166240311 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f457000c980 con 0x7f45781012a0 2023-07-20T18:54:51.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.695+0000 7f456e7fc700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45640a1150 0x7f45640a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:54:51.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.696+0000 7f456e7fc700 1 -- 172.21.15.88:0/166240311 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f45700cb2f0 con 0x7f45781012a0 2023-07-20T18:54:51.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.696+0000 7f457d377700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45640a1150 0x7f45640a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:54:51.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.696+0000 7f457d377700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45640a1150 0x7f45640a3610 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4578100760 tx=0x7f4574009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:54:51.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.699+0000 7f456e7fc700 1 -- 172.21.15.88:0/166240311 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4570095cd0 con 0x7f45781012a0 2023-07-20T18:54:51.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:54:51.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:54:51.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:54:51.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi057 2023-07-20T18:54:51.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:51.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:54:51.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi057 to 8636M 2023-07-20T18:54:51.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:51.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:54:51.836 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:54:51.836 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:51 smithi192 ceph-mon[129200]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T18:54:51.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:54:51.896+0000 7f457f5db700 1 -- 172.21.15.88:0/166240311 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4578060bb0 con 0x7f45640a1150 2023-07-20T18:54:52.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:52 smithi057 ceph-mon[126340]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:54:52.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:52 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:54:52.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:52 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:54:52.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:52 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:52.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:52 smithi057 ceph-mon[126340]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T18:54:52.754 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:52 smithi088 ceph-mon[128823]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:54:52.754 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:52 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:54:52.754 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:52 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:54:52.754 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:52 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:52.754 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:52 smithi088 ceph-mon[128823]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T18:54:52.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:52 smithi192 ceph-mon[129200]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:54:52.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:52 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:54:52.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:52 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:54:52.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:52 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:54:52.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:52 smithi192 ceph-mon[129200]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T18:54:53.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:53 smithi057 ceph-mon[126340]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:54:53.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:53 smithi192 ceph-mon[129200]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:54:53.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:53 smithi088 ceph-mon[128823]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:54:54.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:54 smithi057 ceph-mon[126340]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T18:54:54.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:54 smithi057 ceph-mon[126340]: from='client.? 172.21.15.88:0/75093197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d1dfa-1cff-4f0a-ae72-22bd522ed696"}]: dispatch 2023-07-20T18:54:54.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:54 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d1dfa-1cff-4f0a-ae72-22bd522ed696"}]: dispatch 2023-07-20T18:54:54.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:54 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c8d1dfa-1cff-4f0a-ae72-22bd522ed696"}]': finished 2023-07-20T18:54:54.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:54 smithi057 ceph-mon[126340]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T18:54:54.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:54 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:54:54.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:54 smithi192 ceph-mon[129200]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T18:54:54.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:54 smithi192 ceph-mon[129200]: from='client.? 172.21.15.88:0/75093197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d1dfa-1cff-4f0a-ae72-22bd522ed696"}]: dispatch 2023-07-20T18:54:54.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:54 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d1dfa-1cff-4f0a-ae72-22bd522ed696"}]: dispatch 2023-07-20T18:54:54.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:54 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c8d1dfa-1cff-4f0a-ae72-22bd522ed696"}]': finished 2023-07-20T18:54:54.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:54 smithi192 ceph-mon[129200]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T18:54:54.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:54 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:54:54.876 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:54 smithi088 ceph-mon[128823]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T18:54:54.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:54 smithi088 ceph-mon[128823]: from='client.? 172.21.15.88:0/75093197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d1dfa-1cff-4f0a-ae72-22bd522ed696"}]: dispatch 2023-07-20T18:54:54.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:54 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d1dfa-1cff-4f0a-ae72-22bd522ed696"}]: dispatch 2023-07-20T18:54:54.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:54 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c8d1dfa-1cff-4f0a-ae72-22bd522ed696"}]': finished 2023-07-20T18:54:54.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:54 smithi088 ceph-mon[128823]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T18:54:54.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:54 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:54:55.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:55 smithi057 ceph-mon[126340]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T18:54:55.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:55 smithi057 ceph-mon[126340]: from='client.? 172.21.15.88:0/3163191819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:54:55.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:55 smithi192 ceph-mon[129200]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T18:54:55.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:55 smithi192 ceph-mon[129200]: from='client.? 172.21.15.88:0/3163191819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:54:55.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:55 smithi088 ceph-mon[128823]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T18:54:55.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:55 smithi088 ceph-mon[128823]: from='client.? 172.21.15.88:0/3163191819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:54:58.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:57 smithi192 ceph-mon[129200]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2023-07-20T18:54:58.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:57 smithi088 ceph-mon[128823]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2023-07-20T18:54:58.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:57 smithi057 ceph-mon[126340]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2023-07-20T18:54:59.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:54:59 smithi057 ceph-mon[126340]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-07-20T18:55:00.060 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:54:59 smithi088 ceph-mon[128823]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-07-20T18:55:00.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:54:59 smithi192 ceph-mon[129200]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-07-20T18:55:00.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T18:55:00.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:01.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T18:55:01.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:01.088 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T18:55:01.088 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:01.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:01 smithi057 ceph-mon[126340]: Deploying daemon osd.4 on smithi088 2023-07-20T18:55:01.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:01 smithi057 ceph-mon[126340]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T18:55:01.999 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:01 smithi088 ceph-mon[128823]: Deploying daemon osd.4 on smithi088 2023-07-20T18:55:02.000 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:01 smithi088 ceph-mon[128823]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T18:55:02.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:01 smithi192 ceph-mon[129200]: Deploying daemon osd.4 on smithi088 2023-07-20T18:55:02.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:01 smithi192 ceph-mon[129200]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T18:55:03.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:03 smithi088 ceph-mon[128823]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:03.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:03 smithi057 ceph-mon[126340]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:04.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:03 smithi192 ceph-mon[129200]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:04.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:04.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:04.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:05.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:05.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:05.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:05.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:05.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:05.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:05.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:05 smithi057 ceph-mon[126340]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:06.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:05 smithi192 ceph-mon[129200]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:06.111 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:05 smithi088 ceph-mon[128823]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:07.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.956+0000 7f456e7fc700 1 -- 172.21.15.88:0/166240311 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4578060bb0 con 0x7f45640a1150 2023-07-20T18:55:07.001 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 4 on host 'smithi088' 2023-07-20T18:55:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 -- 172.21.15.88:0/166240311 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45640a1150 msgr2=0x7f45640a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45640a1150 0x7f45640a3610 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4578100760 tx=0x7f4574009190 comp rx=0 tx=0).stop 2023-07-20T18:55:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 -- 172.21.15.88:0/166240311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45781012a0 msgr2=0x7f457819d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45781012a0 0x7f457819d9d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f457000ada0 tx=0x7f457000c330 comp rx=0 tx=0).stop 2023-07-20T18:55:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 -- 172.21.15.88:0/166240311 shutdown_connections 2023-07-20T18:55:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45640a1150 0x7f45640a3610 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45780ff700 0x7f4578068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45781012a0 0x7f457819d9d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 --2- 172.21.15.88:0/166240311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4578100900 0x7f457819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 -- 172.21.15.88:0/166240311 >> 172.21.15.88:0/166240311 conn(0x7f45780fac80 msgr2=0x7f4578103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:55:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 -- 172.21.15.88:0/166240311 shutdown_connections 2023-07-20T18:55:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:06.958+0000 7f457f5db700 1 -- 172.21.15.88:0/166240311 wait complete. 2023-07-20T18:55:07.352 DEBUG:teuthology.orchestra.run.smithi088:osd.4> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.4.service 2023-07-20T18:55:07.354 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_3... 2023-07-20T18:55:07.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T18:55:07.394 INFO:journalctl@ceph.osd.4.smithi088.stdout:-- Logs begin at Thu 2023-07-20 18:25:07 UTC. -- 2023-07-20T18:55:07.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:07.627 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:07.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:07 smithi088 ceph-mon[128823]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:07.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:07.628 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:07.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:07.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:07 smithi192 ceph-mon[129200]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:07.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:07.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.926 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:55:07.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:07.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:07.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:07 smithi057 ceph-mon[126340]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:07.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:07.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:07.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:08.576 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 20 18:55:08 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-4[139332]: 2023-07-20T18:55:08.334+0000 7f8cd779d640 -1 osd.4 0 log_to_monitors true 2023-07-20T18:55:08.725 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:08 smithi192 ceph-mon[129200]: from='osd.4 [v2:172.21.15.88:6808/3911525610,v1:172.21.15.88:6809/3911525610]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T18:55:08.725 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:08 smithi192 ceph-mon[129200]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T18:55:08.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:08 smithi088 ceph-mon[128823]: from='osd.4 [v2:172.21.15.88:6808/3911525610,v1:172.21.15.88:6809/3911525610]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T18:55:08.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:08 smithi088 ceph-mon[128823]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T18:55:08.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:08 smithi057 ceph-mon[126340]: from='osd.4 [v2:172.21.15.88:6808/3911525610,v1:172.21.15.88:6809/3911525610]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T18:55:08.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:08 smithi057 ceph-mon[126340]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T18:55:09.410 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T18:55:09.410 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T18:55:09.410 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-07-20T18:55:09.410 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-07-20T18:55:09.410 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110038 s, 953 MB/s 2023-07-20T18:55:09.410 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-07-20T18:55:09.764 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:09 smithi192 ceph-mon[129200]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T18:55:09.764 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:09 smithi192 ceph-mon[129200]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T18:55:09.764 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:09.764 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:09 smithi192 ceph-mon[129200]: from='osd.4 [v2:172.21.15.88:6808/3911525610,v1:172.21.15.88:6809/3911525610]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:55:09.764 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:09 smithi192 ceph-mon[129200]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:55:09.764 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:09 smithi192 ceph-mon[129200]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:09.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2023-07-20T18:55:09.877 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 20 18:55:09 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-4[139332]: 2023-07-20T18:55:09.550+0000 7f8ccce15700 -1 osd.4 0 waiting for initial osdmap 2023-07-20T18:55:09.877 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 20 18:55:09 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-4[139332]: 2023-07-20T18:55:09.555+0000 7f8cc6406700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:55:09.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:09 smithi088 ceph-mon[128823]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T18:55:09.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:09 smithi088 ceph-mon[128823]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T18:55:09.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:09.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:09 smithi088 ceph-mon[128823]: from='osd.4 [v2:172.21.15.88:6808/3911525610,v1:172.21.15.88:6809/3911525610]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:55:09.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:09 smithi088 ceph-mon[128823]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:55:09.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:09 smithi088 ceph-mon[128823]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:09.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:09 smithi057 ceph-mon[126340]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T18:55:09.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:09 smithi057 ceph-mon[126340]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T18:55:09.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:09.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:09 smithi057 ceph-mon[126340]: from='osd.4 [v2:172.21.15.88:6808/3911525610,v1:172.21.15.88:6809/3911525610]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:55:09.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:09 smithi057 ceph-mon[126340]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:55:09.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:09 smithi057 ceph-mon[126340]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:10.317 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:55:10.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:55:10.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T18:55:10.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:10.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:10.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi088 2023-07-20T18:55:10.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:10.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:10.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi088 to 8584M 2023-07-20T18:55:10.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:10.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:10.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:10 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:55:10.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T18:55:10.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:10.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:10.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi088 2023-07-20T18:55:10.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:10.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:10.879 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi088 to 8584M 2023-07-20T18:55:10.879 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.879 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:10.879 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:10.879 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:10 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:55:10.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T18:55:10.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:10.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:10.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi088 2023-07-20T18:55:10.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:10.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:10.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi088 to 8584M 2023-07-20T18:55:10.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:10.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:10.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:10.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:10 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:11.521 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.477+0000 7f228bb4f700 1 -- 172.21.15.192:0/1927268395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22840ff700 msgr2=0x7f22840ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:11.522 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.477+0000 7f228bb4f700 1 --2- 172.21.15.192:0/1927268395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22840ff700 0x7f22840ffb20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2274009990 tx=0x7f2274009ca0 comp rx=0 tx=0).stop 2023-07-20T18:55:11.522 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.477+0000 7f228bb4f700 1 -- 172.21.15.192:0/1927268395 shutdown_connections 2023-07-20T18:55:11.522 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.477+0000 7f228bb4f700 1 --2- 172.21.15.192:0/1927268395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22840ff700 0x7f22840ffb20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:11.522 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.477+0000 7f228bb4f700 1 --2- 172.21.15.192:0/1927268395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22841012a0 0x7f2284107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:11.522 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.477+0000 7f228bb4f700 1 --2- 172.21.15.192:0/1927268395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2284100900 0x7f2284100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:11.522 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.477+0000 7f228bb4f700 1 -- 172.21.15.192:0/1927268395 >> 172.21.15.192:0/1927268395 conn(0x7f22840fac80 msgr2=0x7f22840fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:55:11.523 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f228bb4f700 1 -- 172.21.15.192:0/1927268395 shutdown_connections 2023-07-20T18:55:11.523 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f228bb4f700 1 -- 172.21.15.192:0/1927268395 wait complete. 2023-07-20T18:55:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f228bb4f700 1 Processor -- start 2023-07-20T18:55:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f228bb4f700 1 -- start start 2023-07-20T18:55:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f228bb4f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22840ff700 0x7f2284195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f228bb4f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2284100900 0x7f2284195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f228bb4f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22841012a0 0x7f228419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f228bb4f700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228419d0d0 con 0x7f22841012a0 2023-07-20T18:55:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f228bb4f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228419d210 con 0x7f2284100900 2023-07-20T18:55:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f228bb4f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228419d3a0 con 0x7f22840ff700 2023-07-20T18:55:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f22898eb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22840ff700 0x7f2284195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:55:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f22898eb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22840ff700 0x7f2284195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:48212/0 (socket says 172.21.15.192:48212) 2023-07-20T18:55:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.478+0000 7f22898eb700 1 -- 172.21.15.192:0/105374709 learned_addr learned my addr 172.21.15.192:0/105374709 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-07-20T18:55:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f22890ea700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2284100900 0x7f2284195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:55:11.526 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f22898eb700 1 -- 172.21.15.192:0/105374709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2284100900 msgr2=0x7f2284195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:11.527 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f22898eb700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2284100900 0x7f2284195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:11.527 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f22898eb700 1 -- 172.21.15.192:0/105374709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22841012a0 msgr2=0x7f228419cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:11.527 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f22898eb700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22841012a0 0x7f228419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:11.527 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f22898eb700 1 -- 172.21.15.192:0/105374709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22740095f0 con 0x7f22840ff700 2023-07-20T18:55:11.527 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f22898eb700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22840ff700 0x7f2284195540 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f22740098e0 tx=0x7f227400f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:55:11.527 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f227affd700 1 -- 172.21.15.192:0/105374709 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227401d070 con 0x7f22840ff700 2023-07-20T18:55:11.527 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f228bb4f700 1 -- 172.21.15.192:0/105374709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228419d620 con 0x7f22840ff700 2023-07-20T18:55:11.528 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f227affd700 1 -- 172.21.15.192:0/105374709 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f227400fc20 con 0x7f22840ff700 2023-07-20T18:55:11.528 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f227affd700 1 -- 172.21.15.192:0/105374709 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22740173f0 con 0x7f22840ff700 2023-07-20T18:55:11.528 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.479+0000 7f228bb4f700 1 -- 172.21.15.192:0/105374709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228419dab0 con 0x7f22840ff700 2023-07-20T18:55:11.528 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.480+0000 7f228bb4f700 1 -- 172.21.15.192:0/105374709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2284106ca0 con 0x7f22840ff700 2023-07-20T18:55:11.528 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.481+0000 7f227affd700 1 -- 172.21.15.192:0/105374709 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2274017550 con 0x7f22840ff700 2023-07-20T18:55:11.529 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.481+0000 7f227affd700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f22700a1150 0x7f22700a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:11.529 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.482+0000 7f227affd700 1 -- 172.21.15.192:0/105374709 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f22740ccba0 con 0x7f22840ff700 2023-07-20T18:55:11.529 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.482+0000 7f22890ea700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f22700a1150 0x7f22700a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:55:11.529 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.482+0000 7f22890ea700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f22700a1150 0x7f22700a3610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2280005950 tx=0x7f22800058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:55:11.530 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.486+0000 7f227affd700 1 -- 172.21.15.192:0/105374709 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f22740973f0 con 0x7f22840ff700 2023-07-20T18:55:11.724 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:11.680+0000 7f228bb4f700 1 -- 172.21.15.192:0/105374709 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2284060bb0 con 0x7f22700a1150 2023-07-20T18:55:11.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:11 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:55:11.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:11 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:55:11.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:11 smithi088 ceph-mon[128823]: osd.4 [v2:172.21.15.88:6808/3911525610,v1:172.21.15.88:6809/3911525610] boot 2023-07-20T18:55:11.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:11 smithi088 ceph-mon[128823]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T18:55:11.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:11.878 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:11 smithi088 ceph-mon[128823]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:11.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:11 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:55:11.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:11 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:55:11.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:11 smithi057 ceph-mon[126340]: osd.4 [v2:172.21.15.88:6808/3911525610,v1:172.21.15.88:6809/3911525610] boot 2023-07-20T18:55:11.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:11 smithi057 ceph-mon[126340]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T18:55:11.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:11.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:11 smithi057 ceph-mon[126340]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:12.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:11 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:55:12.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:11 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:55:12.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:11 smithi192 ceph-mon[129200]: osd.4 [v2:172.21.15.88:6808/3911525610,v1:172.21.15.88:6809/3911525610] boot 2023-07-20T18:55:12.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:11 smithi192 ceph-mon[129200]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T18:55:12.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T18:55:12.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:11 smithi192 ceph-mon[129200]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T18:55:12.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:12 smithi192 ceph-mon[129200]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T18:55:12.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:12 smithi192 ceph-mon[129200]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:55:12.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:12 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:55:12.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:12 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:55:12.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:12 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:12.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:12 smithi088 ceph-mon[128823]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T18:55:12.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:12 smithi088 ceph-mon[128823]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:55:12.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:12 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:55:12.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:12 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:55:12.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:12 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:12.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:12 smithi057 ceph-mon[126340]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T18:55:12.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:12 smithi057 ceph-mon[126340]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:55:12.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:12 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:55:12.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:12 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:55:12.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:12 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:13.877 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:13 smithi088 ceph-mon[128823]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:13.979 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:13 smithi057 ceph-mon[126340]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:14.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:13 smithi192 ceph-mon[129200]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:14.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:14 smithi057 ceph-mon[126340]: from='client.? 172.21.15.192:0/1775045860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a2a56ab-2af9-4176-ba4d-b2138f0e7c75"}]: dispatch 2023-07-20T18:55:14.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:14 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a2a56ab-2af9-4176-ba4d-b2138f0e7c75"}]: dispatch 2023-07-20T18:55:14.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:14 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a2a56ab-2af9-4176-ba4d-b2138f0e7c75"}]': finished 2023-07-20T18:55:14.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:14 smithi057 ceph-mon[126340]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T18:55:14.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:14 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:15.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:14 smithi192 ceph-mon[129200]: from='client.? 172.21.15.192:0/1775045860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a2a56ab-2af9-4176-ba4d-b2138f0e7c75"}]: dispatch 2023-07-20T18:55:15.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:14 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a2a56ab-2af9-4176-ba4d-b2138f0e7c75"}]: dispatch 2023-07-20T18:55:15.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:14 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a2a56ab-2af9-4176-ba4d-b2138f0e7c75"}]': finished 2023-07-20T18:55:15.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:14 smithi192 ceph-mon[129200]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T18:55:15.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:14 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:15.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:14 smithi088 ceph-mon[128823]: from='client.? 172.21.15.192:0/1775045860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a2a56ab-2af9-4176-ba4d-b2138f0e7c75"}]: dispatch 2023-07-20T18:55:15.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:14 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a2a56ab-2af9-4176-ba4d-b2138f0e7c75"}]: dispatch 2023-07-20T18:55:15.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:14 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a2a56ab-2af9-4176-ba4d-b2138f0e7c75"}]': finished 2023-07-20T18:55:15.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:14 smithi088 ceph-mon[128823]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T18:55:15.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:14 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:15.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:15 smithi057 ceph-mon[126340]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:15.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:15 smithi057 ceph-mon[126340]: from='client.? 172.21.15.192:0/3782525000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:55:16.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:15 smithi192 ceph-mon[129200]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:16.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:15 smithi192 ceph-mon[129200]: from='client.? 172.21.15.192:0/3782525000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:55:16.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:15 smithi088 ceph-mon[128823]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:16.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:15 smithi088 ceph-mon[128823]: from='client.? 172.21.15.192:0/3782525000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:55:17.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:17 smithi057 ceph-mon[126340]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:18.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:17 smithi192 ceph-mon[129200]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:18.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:17 smithi088 ceph-mon[128823]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:19.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:19 smithi057 ceph-mon[126340]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:20.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:19 smithi192 ceph-mon[129200]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:20.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:19 smithi088 ceph-mon[128823]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:20.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T18:55:20.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:21.011 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T18:55:21.012 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:21.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T18:55:21.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:21.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:21 smithi057 ceph-mon[126340]: Deploying daemon osd.5 on smithi192 2023-07-20T18:55:21.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:21 smithi057 ceph-mon[126340]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:22.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:21 smithi192 ceph-mon[129200]: Deploying daemon osd.5 on smithi192 2023-07-20T18:55:22.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:21 smithi192 ceph-mon[129200]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:22.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:21 smithi088 ceph-mon[128823]: Deploying daemon osd.5 on smithi192 2023-07-20T18:55:22.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:21 smithi088 ceph-mon[128823]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:24.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:23 smithi192 ceph-mon[129200]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:24.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:23 smithi088 ceph-mon[128823]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:24.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:23 smithi057 ceph-mon[126340]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:25.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:25.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:25.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:25.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:25 smithi192 ceph-mon[129200]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:25.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:25.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:25.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:25.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:25 smithi057 ceph-mon[126340]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:26.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:26.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:26.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:26.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:25 smithi088 ceph-mon[128823]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:27.382 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2023-07-20T18:55:27.383 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.338+0000 7f227affd700 1 -- 172.21.15.192:0/105374709 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2284060bb0 con 0x7f22700a1150 2023-07-20T18:55:27.384 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 -- 172.21.15.192:0/105374709 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f22700a1150 msgr2=0x7f22700a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:27.384 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f22700a1150 0x7f22700a3610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2280005950 tx=0x7f22800058e0 comp rx=0 tx=0).stop 2023-07-20T18:55:27.385 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 -- 172.21.15.192:0/105374709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22840ff700 msgr2=0x7f2284195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:27.385 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22840ff700 0x7f2284195540 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f22740098e0 tx=0x7f227400f740 comp rx=0 tx=0).stop 2023-07-20T18:55:27.385 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 -- 172.21.15.192:0/105374709 shutdown_connections 2023-07-20T18:55:27.385 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f22700a1150 0x7f22700a3610 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:27.385 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22840ff700 0x7f2284195540 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:27.385 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22841012a0 0x7f228419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:27.385 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 --2- 172.21.15.192:0/105374709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2284100900 0x7f2284195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:27.386 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 -- 172.21.15.192:0/105374709 >> 172.21.15.192:0/105374709 conn(0x7f22840fac80 msgr2=0x7f2284103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:55:27.386 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 -- 172.21.15.192:0/105374709 shutdown_connections 2023-07-20T18:55:27.386 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:55:27.340+0000 7f228bb4f700 1 -- 172.21.15.192:0/105374709 wait complete. 2023-07-20T18:55:27.803 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.5.service 2023-07-20T18:55:27.805 INFO:tasks.cephadm:Deploying osd.6 on smithi057 with /dev/vg_nvme/lv_2... 2023-07-20T18:55:27.806 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T18:55:27.837 INFO:journalctl@ceph.osd.5.smithi192.stdout:-- Logs begin at Thu 2023-07-20 18:24:33 UTC. -- 2023-07-20T18:55:28.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:27 smithi192 ceph-mon[129200]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:28.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:27 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:27 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:27 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:28.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:27 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:28.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:27 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:27 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:27 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:27 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:28.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:27 smithi057 ceph-mon[126340]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:28.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:28.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:28.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:28.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:27 smithi088 ceph-mon[128823]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:28.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:27 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:27 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:27 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:28.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:27 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:28.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:27 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:27 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:27 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:28.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:27 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:55:28.835 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:55:28 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:55:28.598+0000 7fb791198640 -1 osd.5 0 log_to_monitors true 2023-07-20T18:55:29.134 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:28 smithi057 ceph-mon[126340]: from='osd.5 [v2:172.21.15.192:6808/905436397,v1:172.21.15.192:6809/905436397]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T18:55:29.134 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:28 smithi057 ceph-mon[126340]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T18:55:29.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:28 smithi192 ceph-mon[129200]: from='osd.5 [v2:172.21.15.192:6808/905436397,v1:172.21.15.192:6809/905436397]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T18:55:29.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:28 smithi192 ceph-mon[129200]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T18:55:29.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:28 smithi088 ceph-mon[128823]: from='osd.5 [v2:172.21.15.192:6808/905436397,v1:172.21.15.192:6809/905436397]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T18:55:29.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:28 smithi088 ceph-mon[128823]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T18:55:30.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:29 smithi057 ceph-mon[126340]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:30.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:29 smithi057 ceph-mon[126340]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T18:55:30.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:29 smithi057 ceph-mon[126340]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T18:55:30.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:29 smithi057 ceph-mon[126340]: from='osd.5 [v2:172.21.15.192:6808/905436397,v1:172.21.15.192:6809/905436397]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:55:30.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:30.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:29 smithi057 ceph-mon[126340]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:55:30.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:29 smithi057 ceph-mon[126340]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:55:30.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:29 smithi057 ceph-mon[126340]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T18:55:30.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:30.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:29 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:30.335 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:55:29.894+0000 7fb786810700 -1 osd.5 0 waiting for initial osdmap 2023-07-20T18:55:30.335 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:55:29.900+0000 7fb780602700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:55:30.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-mon[129200]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:30.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-mon[129200]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T18:55:30.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-mon[129200]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T18:55:30.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-mon[129200]: from='osd.5 [v2:172.21.15.192:6808/905436397,v1:172.21.15.192:6809/905436397]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:55:30.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:30.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-mon[129200]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:55:30.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-mon[129200]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:55:30.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-mon[129200]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T18:55:30.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:30.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:29 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:30.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:29 smithi088 ceph-mon[128823]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:30.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:29 smithi088 ceph-mon[128823]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T18:55:30.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:29 smithi088 ceph-mon[128823]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T18:55:30.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:29 smithi088 ceph-mon[128823]: from='osd.5 [v2:172.21.15.192:6808/905436397,v1:172.21.15.192:6809/905436397]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:55:30.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:30.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:29 smithi088 ceph-mon[128823]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:55:30.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:29 smithi088 ceph-mon[128823]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:55:30.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:29 smithi088 ceph-mon[128823]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T18:55:30.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:30.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:29 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:30.547 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T18:55:30.547 INFO:teuthology.orchestra.run.smithi057.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T18:55:30.548 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10+0 records in 2023-07-20T18:55:30.548 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2023-07-20T18:55:30.548 INFO:teuthology.orchestra.run.smithi057.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0300711 s, 349 MB/s 2023-07-20T18:55:30.548 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping successful for: 2023-07-20T18:55:30.943 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi057:vg_nvme/lv_2 2023-07-20T18:55:31.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi192 2023-07-20T18:55:31.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:31.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:31.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi192 to 8584M 2023-07-20T18:55:31.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:31.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:31.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: osd.5 [v2:172.21.15.192:6808/905436397,v1:172.21.15.192:6809/905436397] boot 2023-07-20T18:55:31.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T18:55:31.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:31.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi192 2023-07-20T18:55:31.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:31.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:31.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi192 to 8584M 2023-07-20T18:55:31.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:31.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:31.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: osd.5 [v2:172.21.15.192:6808/905436397,v1:172.21.15.192:6809/905436397] boot 2023-07-20T18:55:31.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T18:55:31.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:31.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi192 2023-07-20T18:55:31.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:31.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:31.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi192 to 8584M 2023-07-20T18:55:31.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:31.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:31.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:31.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: osd.5 [v2:172.21.15.192:6808/905436397,v1:172.21.15.192:6809/905436397] boot 2023-07-20T18:55:31.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T18:55:31.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T18:55:31.674 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:55:32.216 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:31 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:55:32.216 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:31 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:55:32.216 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:31 smithi057 ceph-mon[126340]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:32.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:31 smithi057 ceph-mon[126340]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T18:55:32.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:31 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:55:32.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:31 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:55:32.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:31 smithi192 ceph-mon[129200]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:32.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:31 smithi192 ceph-mon[129200]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T18:55:32.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:31 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:55:32.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:31 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:55:32.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:31 smithi088 ceph-mon[128823]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T18:55:32.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:31 smithi088 ceph-mon[128823]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T18:55:33.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.964+0000 7f14a797b700 1 -- 172.21.15.57:0/1099432366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14a0100800 msgr2=0x7f14a0100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:33.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.964+0000 7f14a797b700 1 --2- 172.21.15.57:0/1099432366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14a0100800 0x7f14a0100c20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f149c0099c0 tx=0x7f149c009cd0 comp rx=0 tx=0).stop 2023-07-20T18:55:33.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.964+0000 7f14a797b700 1 -- 172.21.15.57:0/1099432366 shutdown_connections 2023-07-20T18:55:33.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.964+0000 7f14a797b700 1 --2- 172.21.15.57:0/1099432366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a01023a0 0x7f14a0108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:33.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.964+0000 7f14a797b700 1 --2- 172.21.15.57:0/1099432366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14a0100800 0x7f14a0100c20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:33.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.964+0000 7f14a797b700 1 --2- 172.21.15.57:0/1099432366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14a0101a00 0x7f14a0101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:33.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.964+0000 7f14a797b700 1 -- 172.21.15.57:0/1099432366 >> 172.21.15.57:0/1099432366 conn(0x7f14a00fbd60 msgr2=0x7f14a00fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:55:33.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.964+0000 7f14a797b700 1 -- 172.21.15.57:0/1099432366 shutdown_connections 2023-07-20T18:55:33.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.964+0000 7f14a797b700 1 -- 172.21.15.57:0/1099432366 wait complete. 2023-07-20T18:55:33.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.965+0000 7f14a797b700 1 Processor -- start 2023-07-20T18:55:33.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a797b700 1 -- start start 2023-07-20T18:55:33.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a797b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14a0100800 0x7f14a01987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:33.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a797b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14a0101a00 0x7f14a0198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:33.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a797b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a01023a0 0x7f14a019fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:33.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a797b700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a01a0370 con 0x7f14a0100800 2023-07-20T18:55:33.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a797b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a01a04b0 con 0x7f14a0101a00 2023-07-20T18:55:33.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a797b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a01a0640 con 0x7f14a01023a0 2023-07-20T18:55:33.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a5717700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14a0100800 0x7f14a01987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:55:33.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a5717700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14a0100800 0x7f14a01987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34942/0 (socket says 172.21.15.57:34942) 2023-07-20T18:55:33.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a5717700 1 -- 172.21.15.57:0/4072925218 learned_addr learned my addr 172.21.15.57:0/4072925218 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:55:33.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.966+0000 7f14a4f16700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14a0101a00 0x7f14a0198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:55:33.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.967+0000 7f14a5f18700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a01023a0 0x7f14a019fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:55:33.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.967+0000 7f14a5717700 1 -- 172.21.15.57:0/4072925218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a01023a0 msgr2=0x7f14a019fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:33.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.967+0000 7f14a5717700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a01023a0 0x7f14a019fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:33.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.967+0000 7f14a5717700 1 -- 172.21.15.57:0/4072925218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14a0101a00 msgr2=0x7f14a0198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:33.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.967+0000 7f14a5717700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14a0101a00 0x7f14a0198d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:33.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.967+0000 7f14a5717700 1 -- 172.21.15.57:0/4072925218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c0095f0 con 0x7f14a0100800 2023-07-20T18:55:33.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.967+0000 7f14a5717700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14a0100800 0x7f14a01987e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f149c00c010 tx=0x7f149c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:55:33.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.967+0000 7f14967fc700 1 -- 172.21.15.57:0/4072925218 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c01d070 con 0x7f14a0100800 2023-07-20T18:55:33.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.967+0000 7f14967fc700 1 -- 172.21.15.57:0/4072925218 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f149c005a00 con 0x7f14a0100800 2023-07-20T18:55:33.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.967+0000 7f14a797b700 1 -- 172.21.15.57:0/4072925218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a01a08c0 con 0x7f14a0100800 2023-07-20T18:55:33.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.968+0000 7f14967fc700 1 -- 172.21.15.57:0/4072925218 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c022d70 con 0x7f14a0100800 2023-07-20T18:55:33.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.968+0000 7f14a797b700 1 -- 172.21.15.57:0/4072925218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a01a0cd0 con 0x7f14a0100800 2023-07-20T18:55:33.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.969+0000 7f14a797b700 1 -- 172.21.15.57:0/4072925218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a010c190 con 0x7f14a0100800 2023-07-20T18:55:33.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.971+0000 7f14967fc700 1 -- 172.21.15.57:0/4072925218 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f149c022470 con 0x7f14a0100800 2023-07-20T18:55:33.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.972+0000 7f14967fc700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f148c0a1150 0x7f148c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:33.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.972+0000 7f14967fc700 1 -- 172.21.15.57:0/4072925218 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f149c0cd5d0 con 0x7f14a0100800 2023-07-20T18:55:33.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.972+0000 7f14a4f16700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f148c0a1150 0x7f148c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:55:33.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.973+0000 7f14a4f16700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f148c0a1150 0x7f148c0a3610 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1490005950 tx=0x7f149000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:55:33.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:32.975+0000 7f14967fc700 1 -- 172.21.15.57:0/4072925218 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f149c097bb0 con 0x7f14a0100800 2023-07-20T18:55:33.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:33.191+0000 7f14a797b700 1 -- 172.21.15.57:0/4072925218 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f14a0108300 con 0x7f148c0a1150 2023-07-20T18:55:34.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:33 smithi057 ceph-mon[126340]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:34.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:33 smithi057 ceph-mon[126340]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T18:55:34.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:55:34.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:55:34.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:34.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:33 smithi057 ceph-mon[126340]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T18:55:34.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:33 smithi192 ceph-mon[129200]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:34.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:33 smithi192 ceph-mon[129200]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T18:55:34.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:55:34.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:55:34.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:34.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:33 smithi192 ceph-mon[129200]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T18:55:34.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:33 smithi088 ceph-mon[128823]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:34.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:33 smithi088 ceph-mon[128823]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T18:55:34.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:55:34.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:55:34.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:34.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:33 smithi088 ceph-mon[128823]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T18:55:35.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:34 smithi057 ceph-mon[126340]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:55:35.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:34 smithi057 ceph-mon[126340]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T18:55:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:34 smithi192 ceph-mon[129200]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:55:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:34 smithi192 ceph-mon[129200]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T18:55:35.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:34 smithi088 ceph-mon[128823]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:55:35.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:34 smithi088 ceph-mon[128823]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T18:55:36.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:35 smithi057 ceph-mon[126340]: pgmap v136: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T18:55:36.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:35 smithi057 ceph-mon[126340]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-20T18:55:36.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:35 smithi192 ceph-mon[129200]: pgmap v136: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T18:55:36.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:35 smithi192 ceph-mon[129200]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-20T18:55:36.377 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:35 smithi088 ceph-mon[128823]: pgmap v136: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T18:55:36.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:35 smithi088 ceph-mon[128823]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-20T18:55:37.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:36 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3715631626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03eafdda-278c-443c-b376-677eeb50262a"}]: dispatch 2023-07-20T18:55:37.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:36 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3715631626' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03eafdda-278c-443c-b376-677eeb50262a"}]': finished 2023-07-20T18:55:37.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:36 smithi057 ceph-mon[126340]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T18:55:37.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:37.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:36 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3715631626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03eafdda-278c-443c-b376-677eeb50262a"}]: dispatch 2023-07-20T18:55:37.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:36 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3715631626' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03eafdda-278c-443c-b376-677eeb50262a"}]': finished 2023-07-20T18:55:37.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:36 smithi192 ceph-mon[129200]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T18:55:37.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:37.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:36 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3715631626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03eafdda-278c-443c-b376-677eeb50262a"}]: dispatch 2023-07-20T18:55:37.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:36 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3715631626' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03eafdda-278c-443c-b376-677eeb50262a"}]': finished 2023-07-20T18:55:37.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:36 smithi088 ceph-mon[128823]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T18:55:37.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:38.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:37 smithi057 ceph-mon[126340]: pgmap v139: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T18:55:38.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:37 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1616250910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:55:38.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:37 smithi192 ceph-mon[129200]: pgmap v139: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T18:55:38.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:37 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1616250910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:55:38.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:37 smithi088 ceph-mon[128823]: pgmap v139: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T18:55:38.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:37 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1616250910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:55:39.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:39 smithi057 ceph-mon[126340]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2023-07-20T18:55:39.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:39 smithi057 ceph-mon[126340]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-20T18:55:39.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:39 smithi057 ceph-mon[126340]: Cluster is now healthy 2023-07-20T18:55:40.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:39 smithi192 ceph-mon[129200]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2023-07-20T18:55:40.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:39 smithi192 ceph-mon[129200]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-20T18:55:40.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:39 smithi192 ceph-mon[129200]: Cluster is now healthy 2023-07-20T18:55:40.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:39 smithi088 ceph-mon[128823]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2023-07-20T18:55:40.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:39 smithi088 ceph-mon[128823]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-20T18:55:40.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:39 smithi088 ceph-mon[128823]: Cluster is now healthy 2023-07-20T18:55:41.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:41 smithi057 ceph-mon[126340]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-20T18:55:42.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:41 smithi192 ceph-mon[129200]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-20T18:55:42.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:41 smithi088 ceph-mon[128823]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-20T18:55:43.980 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:43 smithi057 ceph-mon[126340]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T18:55:44.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:43 smithi192 ceph-mon[129200]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T18:55:44.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:43 smithi088 ceph-mon[128823]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T18:55:44.961 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:44 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T18:55:44.962 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:44 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:44.962 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:44 smithi057 ceph-mon[126340]: Deploying daemon osd.6 on smithi057 2023-07-20T18:55:45.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:44 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T18:55:45.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:44 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:45.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:44 smithi192 ceph-mon[129200]: Deploying daemon osd.6 on smithi057 2023-07-20T18:55:45.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:44 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T18:55:45.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:44 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:45.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:44 smithi088 ceph-mon[128823]: Deploying daemon osd.6 on smithi057 2023-07-20T18:55:46.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:45 smithi192 ceph-mon[129200]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T18:55:46.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:45 smithi088 ceph-mon[128823]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T18:55:46.185 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:45 smithi057 ceph-mon[126340]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T18:55:48.010 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:47 smithi057 ceph-mon[126340]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-07-20T18:55:48.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:47 smithi192 ceph-mon[129200]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-07-20T18:55:48.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:47 smithi088 ceph-mon[128823]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-07-20T18:55:49.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:49 smithi057 ceph-mon[126340]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T18:55:50.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:49 smithi192 ceph-mon[129200]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T18:55:50.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:49 smithi088 ceph-mon[128823]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T18:55:51.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:51.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:51.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:51 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:51.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:51 smithi088 ceph-mon[128823]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:51.406 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:51.406 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:51.406 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:51 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:51.406 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:51 smithi057 ceph-mon[126340]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:51.585 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:51.586 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:51.586 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:51 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:51.586 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:51 smithi192 ceph-mon[129200]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:53.500 INFO:teuthology.orchestra.run.smithi057.stdout:Created osd(s) 6 on host 'smithi057' 2023-07-20T18:55:53.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.454+0000 7f14967fc700 1 -- 172.21.15.57:0/4072925218 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f14a0108300 con 0x7f148c0a1150 2023-07-20T18:55:53.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 -- 172.21.15.57:0/4072925218 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f148c0a1150 msgr2=0x7f148c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:53.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f148c0a1150 0x7f148c0a3610 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1490005950 tx=0x7f149000b410 comp rx=0 tx=0).stop 2023-07-20T18:55:53.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 -- 172.21.15.57:0/4072925218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14a0100800 msgr2=0x7f14a01987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:53.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14a0100800 0x7f14a01987e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f149c00c010 tx=0x7f149c009580 comp rx=0 tx=0).stop 2023-07-20T18:55:53.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 -- 172.21.15.57:0/4072925218 shutdown_connections 2023-07-20T18:55:53.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f148c0a1150 0x7f148c0a3610 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:53.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a01023a0 0x7f14a019fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:53.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14a0100800 0x7f14a01987e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:53.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 --2- 172.21.15.57:0/4072925218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14a0101a00 0x7f14a0198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:53.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 -- 172.21.15.57:0/4072925218 >> 172.21.15.57:0/4072925218 conn(0x7f14a00fbd60 msgr2=0x7f14a00fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:55:53.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 -- 172.21.15.57:0/4072925218 shutdown_connections 2023-07-20T18:55:53.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:55:53.457+0000 7f14a797b700 1 -- 172.21.15.57:0/4072925218 wait complete. 2023-07-20T18:55:53.900 DEBUG:teuthology.orchestra.run.smithi057:osd.6> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.6.service 2023-07-20T18:55:53.904 INFO:tasks.cephadm:Deploying osd.7 on smithi088 with /dev/vg_nvme/lv_2... 2023-07-20T18:55:53.905 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T18:55:53.963 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:53 smithi057 ceph-mon[126340]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:53.963 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:53 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:53.963 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:53 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:53.963 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:53 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:53.964 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:53 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:53.964 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:53 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:53.964 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:53 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:53.964 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:53 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:53.964 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:53 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:53.977 INFO:journalctl@ceph.osd.6.smithi057.stdout:-- Logs begin at Thu 2023-07-20 18:30:45 UTC. -- 2023-07-20T18:55:54.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:53 smithi192 ceph-mon[129200]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:54.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:53 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:54.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:53 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:54.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:53 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:54.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:53 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:54.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:53 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:54.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:53 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:54.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:53 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:54.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:53 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:54.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:53 smithi088 ceph-mon[128823]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:54.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:54.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:54.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:54.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:54.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:54.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:54.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:54.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:53 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:55:54.231 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:55:54 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:55:54.048+0000 7ff5a9055640 -1 osd.6 0 log_to_monitors true 2023-07-20T18:55:54.467 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T18:55:54.956 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:54 smithi057 ceph-mon[126340]: from='osd.6 [v2:172.21.15.57:6818/333894423,v1:172.21.15.57:6819/333894423]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T18:55:54.957 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:54 smithi057 ceph-mon[126340]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T18:55:54.961 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:54 smithi088 ceph-mon[128823]: from='osd.6 [v2:172.21.15.57:6818/333894423,v1:172.21.15.57:6819/333894423]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T18:55:54.961 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:54 smithi088 ceph-mon[128823]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T18:55:55.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:54 smithi192 ceph-mon[129200]: from='osd.6 [v2:172.21.15.57:6818/333894423,v1:172.21.15.57:6819/333894423]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T18:55:55.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:54 smithi192 ceph-mon[129200]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T18:55:55.916 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T18:55:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T18:55:55.917 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-20T18:55:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-20T18:55:55.917 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0156668 s, 669 MB/s 2023-07-20T18:55:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-20T18:55:55.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:55 smithi057 ceph-mon[126340]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:55.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:55 smithi057 ceph-mon[126340]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T18:55:55.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:55 smithi057 ceph-mon[126340]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T18:55:55.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:55 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:55.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:55 smithi057 ceph-mon[126340]: from='osd.6 [v2:172.21.15.57:6818/333894423,v1:172.21.15.57:6819/333894423]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:55:55.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:55 smithi057 ceph-mon[126340]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:55:55.982 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:55:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:55:55.734+0000 7ff59e6cd700 -1 osd.6 0 waiting for initial osdmap 2023-07-20T18:55:55.983 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:55:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:55:55.751+0000 7ff59a4c3700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:55:56.060 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:55 smithi088 ceph-mon[128823]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:56.061 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:55 smithi088 ceph-mon[128823]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T18:55:56.061 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:55 smithi088 ceph-mon[128823]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T18:55:56.061 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:55 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:56.061 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:55 smithi088 ceph-mon[128823]: from='osd.6 [v2:172.21.15.57:6818/333894423,v1:172.21.15.57:6819/333894423]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:55:56.061 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:55 smithi088 ceph-mon[128823]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:55:56.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:55 smithi192 ceph-mon[129200]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:56.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:55 smithi192 ceph-mon[129200]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T18:55:56.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:55 smithi192 ceph-mon[129200]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T18:55:56.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:55 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:56.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:55 smithi192 ceph-mon[129200]: from='osd.6 [v2:172.21.15.57:6818/333894423,v1:172.21.15.57:6819/333894423]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:55:56.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:55 smithi192 ceph-mon[129200]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:55:56.321 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_2 2023-07-20T18:55:56.874 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T18:55:57.058 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:56 smithi088 ceph-mon[128823]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:55:57.059 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:56 smithi088 ceph-mon[128823]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T18:55:57.059 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:56 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:57.059 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:56 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:57.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:56 smithi192 ceph-mon[129200]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:55:57.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:56 smithi192 ceph-mon[129200]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T18:55:57.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:56 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:57.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:56 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:57.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:56 smithi057 ceph-mon[126340]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:55:57.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:56 smithi057 ceph-mon[126340]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T18:55:57.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:56 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:57.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:56 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 -- 172.21.15.88:0/3388591324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d41012d0 msgr2=0x7f15d4107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 --2- 172.21.15.88:0/3388591324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d41012d0 0x7f15d4107b80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f15cc00b0d0 tx=0x7f15cc00b3e0 comp rx=0 tx=0).stop 2023-07-20T18:55:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 -- 172.21.15.88:0/3388591324 shutdown_connections 2023-07-20T18:55:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 --2- 172.21.15.88:0/3388591324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d41012d0 0x7f15d4107b80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 --2- 172.21.15.88:0/3388591324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15d40ff730 0x7f15d40ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 --2- 172.21.15.88:0/3388591324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d4100930 0x7f15d4100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 -- 172.21.15.88:0/3388591324 >> 172.21.15.88:0/3388591324 conn(0x7f15d40facf0 msgr2=0x7f15d40fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:55:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 -- 172.21.15.88:0/3388591324 shutdown_connections 2023-07-20T18:55:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 -- 172.21.15.88:0/3388591324 wait complete. 2023-07-20T18:55:58.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 Processor -- start 2023-07-20T18:55:58.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 -- start start 2023-07-20T18:55:58.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d40ff730 0x7f15d4196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:58.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15d4100930 0x7f15d4196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:58.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d41012d0 0x7f15d419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:58.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d419e3b0 con 0x7f15d4100930 2023-07-20T18:55:58.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d419e4f0 con 0x7f15d41012d0 2023-07-20T18:55:58.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.977+0000 7f15dc708700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d419e680 con 0x7f15d40ff730 2023-07-20T18:55:58.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15daca5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d41012d0 0x7f15d419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:55:58.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15daca5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d41012d0 0x7f15d419dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46112/0 (socket says 172.21.15.88:46112) 2023-07-20T18:55:58.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15daca5700 1 -- 172.21.15.88:0/450373309 learned_addr learned my addr 172.21.15.88:0/450373309 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-20T18:55:58.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15da4a4700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d40ff730 0x7f15d4196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:55:58.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15daca5700 1 -- 172.21.15.88:0/450373309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d40ff730 msgr2=0x7f15d4196820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:58.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15daca5700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d40ff730 0x7f15d4196820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:58.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15daca5700 1 -- 172.21.15.88:0/450373309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15d4100930 msgr2=0x7f15d4196d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:55:58.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15daca5700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15d4100930 0x7f15d4196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:55:58.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15daca5700 1 -- 172.21.15.88:0/450373309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15cc009d00 con 0x7f15d41012d0 2023-07-20T18:55:58.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15da4a4700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d40ff730 0x7f15d4196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T18:55:58.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15daca5700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d41012d0 0x7f15d419dde0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f15cc007b60 tx=0x7f15cc009430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:55:58.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15cb7fe700 1 -- 172.21.15.88:0/450373309 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc00e040 con 0x7f15d41012d0 2023-07-20T18:55:58.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15cb7fe700 1 -- 172.21.15.88:0/450373309 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f15cc01d960 con 0x7f15d41012d0 2023-07-20T18:55:58.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15dc708700 1 -- 172.21.15.88:0/450373309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d419e900 con 0x7f15d41012d0 2023-07-20T18:55:58.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15cb7fe700 1 -- 172.21.15.88:0/450373309 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc01bac0 con 0x7f15d41012d0 2023-07-20T18:55:58.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.978+0000 7f15dc708700 1 -- 172.21.15.88:0/450373309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d419edf0 con 0x7f15d41012d0 2023-07-20T18:55:58.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.979+0000 7f15dc708700 1 -- 172.21.15.88:0/450373309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d41077c0 con 0x7f15d41012d0 2023-07-20T18:55:58.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.980+0000 7f15cb7fe700 1 -- 172.21.15.88:0/450373309 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f15cc01dad0 con 0x7f15d41012d0 2023-07-20T18:55:58.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.981+0000 7f15cb7fe700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f15c00a1150 0x7f15c00a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:55:58.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.981+0000 7f15cb7fe700 1 -- 172.21.15.88:0/450373309 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f15cc0cdba0 con 0x7f15d41012d0 2023-07-20T18:55:58.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.981+0000 7f15da4a4700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f15c00a1150 0x7f15c00a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:55:58.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.982+0000 7f15da4a4700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f15c00a1150 0x7f15c00a3610 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f15d4100790 tx=0x7f15d0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:55:58.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:57.984+0000 7f15cb7fe700 1 -- 172.21.15.88:0/450373309 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f15cc098090 con 0x7f15d41012d0 2023-07-20T18:55:58.085 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:55:58.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:55:58.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:58.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: osd.6 [v2:172.21.15.57:6818/333894423,v1:172.21.15.57:6819/333894423] boot 2023-07-20T18:55:58.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T18:55:58.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:58.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi057 2023-07-20T18:55:58.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:58.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:58.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:58.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi057 to 5757M 2023-07-20T18:55:58.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:58.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:58.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.128 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:55:58.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:55:58.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:58.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: osd.6 [v2:172.21.15.57:6818/333894423,v1:172.21.15.57:6819/333894423] boot 2023-07-20T18:55:58.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T18:55:58.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:58.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi057 2023-07-20T18:55:58.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:58.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:58.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:58.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi057 to 5757M 2023-07-20T18:55:58.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:58.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:58.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:55:58.177+0000 7f15dc708700 1 -- 172.21.15.88:0/450373309 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f15d4060bb0 con 0x7f15c00a1150 2023-07-20T18:55:58.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:55:58.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:55:58.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T18:55:58.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: osd.6 [v2:172.21.15.57:6818/333894423,v1:172.21.15.57:6819/333894423] boot 2023-07-20T18:55:58.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T18:55:58.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T18:55:58.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi057 2023-07-20T18:55:58.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:58.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:58.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:55:58.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi057 to 5757M 2023-07-20T18:55:58.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:58.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:58.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:55:58.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:55:59.193 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:58 smithi088 ceph-mon[128823]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T18:55:59.194 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:55:59.194 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:55:59.194 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:59.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:58 smithi057 ceph-mon[126340]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T18:55:59.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:55:59.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:55:59.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:55:59.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:58 smithi192 ceph-mon[129200]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T18:55:59.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:55:59.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:55:59.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:00.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:59 smithi057 ceph-mon[126340]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:56:00.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:55:59 smithi057 ceph-mon[126340]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:00.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:59 smithi192 ceph-mon[129200]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:56:00.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:55:59 smithi192 ceph-mon[129200]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:00.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:59 smithi088 ceph-mon[128823]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:56:00.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:55:59 smithi088 ceph-mon[128823]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:01.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:00 smithi057 ceph-mon[126340]: from='client.? 172.21.15.88:0/1247105685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f01254e-be73-41c8-b979-3b6a7534e2e1"}]: dispatch 2023-07-20T18:56:01.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:00 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f01254e-be73-41c8-b979-3b6a7534e2e1"}]: dispatch 2023-07-20T18:56:01.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:00 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f01254e-be73-41c8-b979-3b6a7534e2e1"}]': finished 2023-07-20T18:56:01.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:00 smithi057 ceph-mon[126340]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T18:56:01.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:00 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:01.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:00 smithi192 ceph-mon[129200]: from='client.? 172.21.15.88:0/1247105685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f01254e-be73-41c8-b979-3b6a7534e2e1"}]: dispatch 2023-07-20T18:56:01.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:00 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f01254e-be73-41c8-b979-3b6a7534e2e1"}]: dispatch 2023-07-20T18:56:01.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:00 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f01254e-be73-41c8-b979-3b6a7534e2e1"}]': finished 2023-07-20T18:56:01.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:00 smithi192 ceph-mon[129200]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T18:56:01.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:00 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:01.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:00 smithi088 ceph-mon[128823]: from='client.? 172.21.15.88:0/1247105685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f01254e-be73-41c8-b979-3b6a7534e2e1"}]: dispatch 2023-07-20T18:56:01.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:00 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f01254e-be73-41c8-b979-3b6a7534e2e1"}]: dispatch 2023-07-20T18:56:01.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:00 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f01254e-be73-41c8-b979-3b6a7534e2e1"}]': finished 2023-07-20T18:56:01.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:00 smithi088 ceph-mon[128823]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T18:56:01.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:00 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:02.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:01 smithi057 ceph-mon[126340]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:02.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:01 smithi057 ceph-mon[126340]: from='client.? 172.21.15.88:0/1585572218' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:56:02.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:01 smithi192 ceph-mon[129200]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:02.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:01 smithi192 ceph-mon[129200]: from='client.? 172.21.15.88:0/1585572218' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:56:02.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:01 smithi088 ceph-mon[128823]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:02.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:01 smithi088 ceph-mon[128823]: from='client.? 172.21.15.88:0/1585572218' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:56:04.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:03 smithi057 ceph-mon[126340]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:04.259 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:03 smithi088 ceph-mon[128823]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:04.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:03 smithi192 ceph-mon[129200]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:06.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:05 smithi057 ceph-mon[126340]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:06.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:05 smithi192 ceph-mon[129200]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:06.378 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:05 smithi088 ceph-mon[128823]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:07.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T18:56:07.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:07.289 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T18:56:07.289 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:07.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T18:56:07.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:08.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:07 smithi057 ceph-mon[126340]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:08.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:07 smithi057 ceph-mon[126340]: Deploying daemon osd.7 on smithi088 2023-07-20T18:56:08.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:07 smithi192 ceph-mon[129200]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:08.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:07 smithi192 ceph-mon[129200]: Deploying daemon osd.7 on smithi088 2023-07-20T18:56:08.371 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:07 smithi088 ceph-mon[128823]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:08.371 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:07 smithi088 ceph-mon[128823]: Deploying daemon osd.7 on smithi088 2023-07-20T18:56:10.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:09 smithi057 ceph-mon[126340]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:10.336 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:09 smithi192 ceph-mon[129200]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:10.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:09 smithi088 ceph-mon[128823]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:11.879 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:11.879 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:11.880 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:11.880 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:11 smithi088 ceph-mon[128823]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:11.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:11.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:11.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:11.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:11 smithi057 ceph-mon[126340]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:12.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:12.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:12.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:12.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:11 smithi192 ceph-mon[129200]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:13.286 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 7 on host 'smithi088' 2023-07-20T18:56:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.239+0000 7f15cb7fe700 1 -- 172.21.15.88:0/450373309 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f15d4060bb0 con 0x7f15c00a1150 2023-07-20T18:56:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 -- 172.21.15.88:0/450373309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f15c00a1150 msgr2=0x7f15c00a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:56:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f15c00a1150 0x7f15c00a3610 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f15d4100790 tx=0x7f15d0008040 comp rx=0 tx=0).stop 2023-07-20T18:56:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 -- 172.21.15.88:0/450373309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d41012d0 msgr2=0x7f15d419dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:56:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d41012d0 0x7f15d419dde0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f15cc007b60 tx=0x7f15cc009430 comp rx=0 tx=0).stop 2023-07-20T18:56:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 -- 172.21.15.88:0/450373309 shutdown_connections 2023-07-20T18:56:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f15c00a1150 0x7f15c00a3610 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d40ff730 0x7f15d4196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d41012d0 0x7f15d419dde0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 --2- 172.21.15.88:0/450373309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15d4100930 0x7f15d4196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 -- 172.21.15.88:0/450373309 >> 172.21.15.88:0/450373309 conn(0x7f15d40facf0 msgr2=0x7f15d40fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:56:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 -- 172.21.15.88:0/450373309 shutdown_connections 2023-07-20T18:56:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:56:13.242+0000 7f15dc708700 1 -- 172.21.15.88:0/450373309 wait complete. 2023-07-20T18:56:13.656 DEBUG:teuthology.orchestra.run.smithi088:osd.7> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.7.service 2023-07-20T18:56:13.659 INFO:tasks.cephadm:Deploying osd.8 on smithi192 with /dev/vg_nvme/lv_2... 2023-07-20T18:56:13.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T18:56:13.688 INFO:journalctl@ceph.osd.7.smithi088.stdout:-- Logs begin at Thu 2023-07-20 18:25:07 UTC. -- 2023-07-20T18:56:14.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:13 smithi192 ceph-mon[129200]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:14.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:14.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:14.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:14.121 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:13 smithi088 ceph-mon[128823]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:14.121 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.121 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.122 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:14.122 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:14.122 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.122 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.122 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.122 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:56:14.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:13 smithi057 ceph-mon[126340]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:14.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:14.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:14.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:14.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:15.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:14 smithi192 ceph-mon[129200]: from='osd.7 [v2:172.21.15.88:6816/3031485551,v1:172.21.15.88:6817/3031485551]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T18:56:15.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:14 smithi192 ceph-mon[129200]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T18:56:15.129 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:56:14 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:56:14.666+0000 7f920a144640 -1 osd.7 0 log_to_monitors true 2023-07-20T18:56:15.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:14 smithi088 ceph-mon[128823]: from='osd.7 [v2:172.21.15.88:6816/3031485551,v1:172.21.15.88:6817/3031485551]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T18:56:15.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:14 smithi088 ceph-mon[128823]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T18:56:15.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:14 smithi057 ceph-mon[126340]: from='osd.7 [v2:172.21.15.88:6816/3031485551,v1:172.21.15.88:6817/3031485551]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T18:56:15.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:14 smithi057 ceph-mon[126340]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T18:56:15.720 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T18:56:15.721 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T18:56:15.721 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-07-20T18:56:15.721 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-07-20T18:56:15.721 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110175 s, 952 MB/s 2023-07-20T18:56:15.721 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-07-20T18:56:16.032 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:16.032 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T18:56:16.032 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T18:56:16.033 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:16.033 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='osd.7 [v2:172.21.15.88:6816/3031485551,v1:172.21.15.88:6817/3031485551]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:56:16.033 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:56:16.033 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:16.033 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:16.033 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:16.033 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:16.033 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:16.034 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:16.034 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:16.034 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:16.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2023-07-20T18:56:16.128 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:56:15.750+0000 7f91ff7bc700 -1 osd.7 0 waiting for initial osdmap 2023-07-20T18:56:16.129 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:56:15.758+0000 7f91f8dad700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:56:16.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:16.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T18:56:16.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T18:56:16.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:16.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='osd.7 [v2:172.21.15.88:6816/3031485551,v1:172.21.15.88:6817/3031485551]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:56:16.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:56:16.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:16.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:16.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:16.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:16.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:16.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:16.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:16.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:16.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:16.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T18:56:16.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T18:56:16.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:16.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='osd.7 [v2:172.21.15.88:6816/3031485551,v1:172.21.15.88:6817/3031485551]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:56:16.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:56:16.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:16.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:16.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:16.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:16.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:16.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:16.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:16.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:16.628 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:56:17.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:16 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi088 2023-07-20T18:56:17.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:16 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi088 to 5722M 2023-07-20T18:56:17.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:16 smithi192 ceph-mon[129200]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:56:17.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:16 smithi192 ceph-mon[129200]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T18:56:17.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:16 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:17.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:16 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:17.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:16 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:17.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:16 smithi192 ceph-mon[129200]: osd.7 [v2:172.21.15.88:6816/3031485551,v1:172.21.15.88:6817/3031485551] boot 2023-07-20T18:56:17.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:16 smithi192 ceph-mon[129200]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T18:56:17.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:16 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:17.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:16 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi088 2023-07-20T18:56:17.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:16 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi088 to 5722M 2023-07-20T18:56:17.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:16 smithi088 ceph-mon[128823]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:56:17.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:16 smithi088 ceph-mon[128823]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T18:56:17.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:16 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:17.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:16 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:17.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:16 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:17.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:16 smithi088 ceph-mon[128823]: osd.7 [v2:172.21.15.88:6816/3031485551,v1:172.21.15.88:6817/3031485551] boot 2023-07-20T18:56:17.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:16 smithi088 ceph-mon[128823]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T18:56:17.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:16 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:17.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:16 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi088 2023-07-20T18:56:17.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:16 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi088 to 5722M 2023-07-20T18:56:17.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:16 smithi057 ceph-mon[126340]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:56:17.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:16 smithi057 ceph-mon[126340]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T18:56:17.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:16 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:17.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:16 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:17.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:16 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:17.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:16 smithi057 ceph-mon[126340]: osd.7 [v2:172.21.15.88:6816/3031485551,v1:172.21.15.88:6817/3031485551] boot 2023-07-20T18:56:17.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:16 smithi057 ceph-mon[126340]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T18:56:17.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:16 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T18:56:17.807 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.761+0000 7f664a544700 1 -- 172.21.15.192:0/446652913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66440ff700 msgr2=0x7f66440ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:56:17.808 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.761+0000 7f664a544700 1 --2- 172.21.15.192:0/446652913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66440ff700 0x7f66440ffb20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f662c009990 tx=0x7f662c009ca0 comp rx=0 tx=0).stop 2023-07-20T18:56:17.808 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.761+0000 7f664a544700 1 -- 172.21.15.192:0/446652913 shutdown_connections 2023-07-20T18:56:17.808 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.761+0000 7f664a544700 1 --2- 172.21.15.192:0/446652913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66440ff700 0x7f66440ffb20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:17.808 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.761+0000 7f664a544700 1 --2- 172.21.15.192:0/446652913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66441012a0 0x7f6644107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:17.808 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.761+0000 7f664a544700 1 --2- 172.21.15.192:0/446652913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6644100900 0x7f6644100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:17.809 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.761+0000 7f664a544700 1 -- 172.21.15.192:0/446652913 >> 172.21.15.192:0/446652913 conn(0x7f66440fac80 msgr2=0x7f66440fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:56:17.809 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.761+0000 7f664a544700 1 -- 172.21.15.192:0/446652913 shutdown_connections 2023-07-20T18:56:17.809 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.761+0000 7f664a544700 1 -- 172.21.15.192:0/446652913 wait complete. 2023-07-20T18:56:17.809 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f664a544700 1 Processor -- start 2023-07-20T18:56:17.809 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f664a544700 1 -- start start 2023-07-20T18:56:17.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f664a544700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66440ff700 0x7f66441954f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:56:17.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f664a544700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6644100900 0x7f6644195a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:56:17.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f664a544700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66441012a0 0x7f664419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:56:17.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f664a544700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664419d080 con 0x7f66440ff700 2023-07-20T18:56:17.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f664a544700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664419d1c0 con 0x7f6644100900 2023-07-20T18:56:17.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f664a544700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664419d350 con 0x7f66441012a0 2023-07-20T18:56:17.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f6648ae1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66441012a0 0x7f664419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:56:17.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f6648ae1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66441012a0 0x7f664419cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:48332/0 (socket says 172.21.15.192:48332) 2023-07-20T18:56:17.812 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f6648ae1700 1 -- 172.21.15.192:0/1461710355 learned_addr learned my addr 172.21.15.192:0/1461710355 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-07-20T18:56:17.812 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f66437fe700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6644100900 0x7f6644195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:56:17.812 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.762+0000 7f6643fff700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66440ff700 0x7f66441954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:56:17.812 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f6648ae1700 1 -- 172.21.15.192:0/1461710355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6644100900 msgr2=0x7f6644195a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:56:17.812 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f6648ae1700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6644100900 0x7f6644195a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:17.812 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f6648ae1700 1 -- 172.21.15.192:0/1461710355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66440ff700 msgr2=0x7f66441954f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:56:17.812 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f6648ae1700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66440ff700 0x7f66441954f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:17.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f6648ae1700 1 -- 172.21.15.192:0/1461710355 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c0095f0 con 0x7f66441012a0 2023-07-20T18:56:17.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f6648ae1700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66441012a0 0x7f664419cab0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f663c007f10 tx=0x7f663c00f5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:56:17.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f66417fa700 1 -- 172.21.15.192:0/1461710355 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c003a00 con 0x7f66441012a0 2023-07-20T18:56:17.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f66417fa700 1 -- 172.21.15.192:0/1461710355 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f663c003b60 con 0x7f66441012a0 2023-07-20T18:56:17.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f664a544700 1 -- 172.21.15.192:0/1461710355 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6644068290 con 0x7f66441012a0 2023-07-20T18:56:17.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f66417fa700 1 -- 172.21.15.192:0/1461710355 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c012750 con 0x7f66441012a0 2023-07-20T18:56:17.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.763+0000 7f664a544700 1 -- 172.21.15.192:0/1461710355 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6644068790 con 0x7f66441012a0 2023-07-20T18:56:17.814 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.764+0000 7f664a544700 1 -- 172.21.15.192:0/1461710355 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664404e470 con 0x7f66441012a0 2023-07-20T18:56:17.814 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.766+0000 7f66417fa700 1 -- 172.21.15.192:0/1461710355 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f663c0128b0 con 0x7f66441012a0 2023-07-20T18:56:17.814 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.766+0000 7f66417fa700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66300a11d0 0x7f66300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:56:17.815 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.766+0000 7f66417fa700 1 -- 172.21.15.192:0/1461710355 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f663c0ce150 con 0x7f66441012a0 2023-07-20T18:56:17.815 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.767+0000 7f6643fff700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66300a11d0 0x7f66300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:56:17.815 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.767+0000 7f6643fff700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66300a11d0 0x7f66300a3690 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f662c00b5c0 tx=0x7f662c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:56:17.815 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.769+0000 7f66417fa700 1 -- 172.21.15.192:0/1461710355 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f663c0983e0 con 0x7f66441012a0 2023-07-20T18:56:18.007 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:17.962+0000 7f664a544700 1 -- 172.21.15.192:0/1461710355 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6644068a70 con 0x7f66300a11d0 2023-07-20T18:56:18.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:17 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:56:18.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:17 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:56:18.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:17 smithi192 ceph-mon[129200]: pgmap v166: 1 pgs: 1 unknown; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:18.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:17 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:56:18.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:17 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:56:18.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:17 smithi088 ceph-mon[128823]: pgmap v166: 1 pgs: 1 unknown; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:18.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:17 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:56:18.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:17 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:56:18.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:17 smithi057 ceph-mon[126340]: pgmap v166: 1 pgs: 1 unknown; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T18:56:19.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:18 smithi192 ceph-mon[129200]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T18:56:19.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:18 smithi192 ceph-mon[129200]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:56:19.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:56:19.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:56:19.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:18 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:19.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:18 smithi088 ceph-mon[128823]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T18:56:19.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:18 smithi088 ceph-mon[128823]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:56:19.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:56:19.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:56:19.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:18 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:19.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:18 smithi057 ceph-mon[126340]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T18:56:19.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:18 smithi057 ceph-mon[126340]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:56:19.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:56:19.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:56:19.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:18 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:19.836 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:19 smithi192 ceph-mon[129200]: pgmap v169: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:19.836 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:19 smithi192 ceph-mon[129200]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T18:56:20.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:19 smithi088 ceph-mon[128823]: pgmap v169: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:20.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:19 smithi088 ceph-mon[128823]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T18:56:20.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:19 smithi057 ceph-mon[126340]: pgmap v169: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:20.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:19 smithi057 ceph-mon[126340]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T18:56:21.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:20 smithi192 ceph-mon[129200]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T18:56:21.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:20 smithi192 ceph-mon[129200]: from='client.? 172.21.15.192:0/1524554253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624a150e-f47b-4a76-8785-c818dd65342e"}]: dispatch 2023-07-20T18:56:21.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:20 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624a150e-f47b-4a76-8785-c818dd65342e"}]: dispatch 2023-07-20T18:56:21.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:20 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "624a150e-f47b-4a76-8785-c818dd65342e"}]': finished 2023-07-20T18:56:21.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:20 smithi192 ceph-mon[129200]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T18:56:21.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:21.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:20 smithi088 ceph-mon[128823]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T18:56:21.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:20 smithi088 ceph-mon[128823]: from='client.? 172.21.15.192:0/1524554253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624a150e-f47b-4a76-8785-c818dd65342e"}]: dispatch 2023-07-20T18:56:21.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:20 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624a150e-f47b-4a76-8785-c818dd65342e"}]: dispatch 2023-07-20T18:56:21.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:20 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "624a150e-f47b-4a76-8785-c818dd65342e"}]': finished 2023-07-20T18:56:21.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:20 smithi088 ceph-mon[128823]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T18:56:21.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:21.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:20 smithi057 ceph-mon[126340]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T18:56:21.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:20 smithi057 ceph-mon[126340]: from='client.? 172.21.15.192:0/1524554253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624a150e-f47b-4a76-8785-c818dd65342e"}]: dispatch 2023-07-20T18:56:21.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:20 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624a150e-f47b-4a76-8785-c818dd65342e"}]: dispatch 2023-07-20T18:56:21.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:20 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "624a150e-f47b-4a76-8785-c818dd65342e"}]': finished 2023-07-20T18:56:21.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:20 smithi057 ceph-mon[126340]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T18:56:21.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:22.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:21 smithi192 ceph-mon[129200]: pgmap v173: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:22.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:21 smithi192 ceph-mon[129200]: from='client.? 172.21.15.192:0/3372259698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:56:22.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:21 smithi088 ceph-mon[128823]: pgmap v173: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:22.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:21 smithi088 ceph-mon[128823]: from='client.? 172.21.15.192:0/3372259698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:56:22.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:21 smithi057 ceph-mon[126340]: pgmap v173: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:22.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:21 smithi057 ceph-mon[126340]: from='client.? 172.21.15.192:0/3372259698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:56:24.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:23 smithi192 ceph-mon[129200]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:24.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:23 smithi088 ceph-mon[128823]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:24.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:23 smithi057 ceph-mon[126340]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:26.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:25 smithi192 ceph-mon[129200]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:26.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:25 smithi088 ceph-mon[128823]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:26.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:25 smithi057 ceph-mon[126340]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:27.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:26 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T18:56:27.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:26 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:27.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:26 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T18:56:27.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:26 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:27.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:26 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T18:56:27.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:26 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:28.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:27 smithi192 ceph-mon[129200]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:28.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:27 smithi192 ceph-mon[129200]: Deploying daemon osd.8 on smithi192 2023-07-20T18:56:28.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:27 smithi088 ceph-mon[128823]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:28.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:27 smithi088 ceph-mon[128823]: Deploying daemon osd.8 on smithi192 2023-07-20T18:56:28.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:27 smithi057 ceph-mon[126340]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:28.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:27 smithi057 ceph-mon[126340]: Deploying daemon osd.8 on smithi192 2023-07-20T18:56:30.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:29 smithi088 ceph-mon[128823]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:30.231 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:29 smithi192 ceph-mon[129200]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:30.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:29 smithi057 ceph-mon[126340]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:32.128 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:31 smithi192 ceph-mon[129200]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:32.129 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:31 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:32.129 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:31 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:32.129 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:31 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:32.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:31 smithi088 ceph-mon[128823]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:32.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:31 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:32.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:31 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:32.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:31 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:32.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:31 smithi057 ceph-mon[126340]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:32.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:31 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:32.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:31 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:32.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:31 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:33.873 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.826+0000 7f66417fa700 1 -- 172.21.15.192:0/1461710355 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6644068a70 con 0x7f66300a11d0 2023-07-20T18:56:33.874 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 8 on host 'smithi192' 2023-07-20T18:56:33.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 -- 172.21.15.192:0/1461710355 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66300a11d0 msgr2=0x7f66300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:56:33.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66300a11d0 0x7f66300a3690 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f662c00b5c0 tx=0x7f662c005fb0 comp rx=0 tx=0).stop 2023-07-20T18:56:33.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 -- 172.21.15.192:0/1461710355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66441012a0 msgr2=0x7f664419cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:56:33.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66441012a0 0x7f664419cab0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f663c007f10 tx=0x7f663c00f5c0 comp rx=0 tx=0).stop 2023-07-20T18:56:33.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 -- 172.21.15.192:0/1461710355 shutdown_connections 2023-07-20T18:56:33.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66300a11d0 0x7f66300a3690 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:33.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66441012a0 0x7f664419cab0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:33.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66440ff700 0x7f66441954f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:33.877 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 --2- 172.21.15.192:0/1461710355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6644100900 0x7f6644195a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:33.877 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 -- 172.21.15.192:0/1461710355 >> 172.21.15.192:0/1461710355 conn(0x7f66440fac80 msgr2=0x7f66440fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:56:33.877 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 -- 172.21.15.192:0/1461710355 shutdown_connections 2023-07-20T18:56:33.877 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:56:33.828+0000 7f664a544700 1 -- 172.21.15.192:0/1461710355 wait complete. 2023-07-20T18:56:34.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:33 smithi192 ceph-mon[129200]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:34.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:34.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:34.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:34.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:33 smithi088 ceph-mon[128823]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:34.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:34.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:34.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:34.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:33 smithi057 ceph-mon[126340]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:34.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:34.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:34.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:34.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:34.321 DEBUG:teuthology.orchestra.run.smithi192:osd.8> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.8.service 2023-07-20T18:56:34.323 INFO:tasks.cephadm:Deploying osd.9 on smithi057 with /dev/vg_nvme/lv_1... 2023-07-20T18:56:34.323 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T18:56:34.356 INFO:journalctl@ceph.osd.8.smithi192.stdout:-- Logs begin at Thu 2023-07-20 18:24:33 UTC. -- 2023-07-20T18:56:35.061 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:56:35.307 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:56:34 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:56:34.996+0000 7f1d7c36a640 -1 osd.8 0 log_to_monitors true 2023-07-20T18:56:36.129 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:35 smithi088 ceph-mon[128823]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:36.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:35 smithi088 ceph-mon[128823]: from='osd.8 [v2:172.21.15.192:6816/918056390,v1:172.21.15.192:6817/918056390]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T18:56:36.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:35 smithi088 ceph-mon[128823]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T18:56:36.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:35 smithi057 ceph-mon[126340]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:36.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:35 smithi057 ceph-mon[126340]: from='osd.8 [v2:172.21.15.192:6816/918056390,v1:172.21.15.192:6817/918056390]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T18:56:36.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:35 smithi057 ceph-mon[126340]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T18:56:36.221 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:35 smithi192 ceph-mon[129200]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:36.221 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:35 smithi192 ceph-mon[129200]: from='osd.8 [v2:172.21.15.192:6816/918056390,v1:172.21.15.192:6817/918056390]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T18:56:36.221 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:35 smithi192 ceph-mon[129200]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T18:56:36.974 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T18:56:36.974 INFO:teuthology.orchestra.run.smithi057.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T18:56:36.975 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10+0 records in 2023-07-20T18:56:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2023-07-20T18:56:36.975 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0573955 s, 183 MB/s 2023-07-20T18:56:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping successful for: 2023-07-20T18:56:37.137 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T18:56:37.138 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T18:56:37.138 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:37.138 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='osd.8 [v2:172.21.15.192:6816/918056390,v1:172.21.15.192:6817/918056390]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:56:37.138 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:56:37.138 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.138 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.138 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:37.139 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:37.139 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:37.139 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.139 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:37.139 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:37.139 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:36 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.337 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:56:36.837+0000 7f1d719e2700 -1 osd.8 0 waiting for initial osdmap 2023-07-20T18:56:37.337 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:56:36.849+0000 7f1d6b7d4700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:56:37.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T18:56:37.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T18:56:37.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:37.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='osd.8 [v2:172.21.15.192:6816/918056390,v1:172.21.15.192:6817/918056390]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:56:37.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:56:37.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:37.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:37.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:37.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:37.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:37.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:36 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T18:56:37.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T18:56:37.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:37.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='osd.8 [v2:172.21.15.192:6816/918056390,v1:172.21.15.192:6817/918056390]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:56:37.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:56:37.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:37.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:37.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:56:37.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:37.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:56:37.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:36 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:37.442 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi057:vg_nvme/lv_1 2023-07-20T18:56:38.178 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:56:38.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:37 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi192 2023-07-20T18:56:38.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:37 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi192 to 5722M 2023-07-20T18:56:38.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:37 smithi057 ceph-mon[126340]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:38.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:37 smithi057 ceph-mon[126340]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:56:38.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:37 smithi057 ceph-mon[126340]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T18:56:38.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:38.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:38.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:37 smithi057 ceph-mon[126340]: osd.8 [v2:172.21.15.192:6816/918056390,v1:172.21.15.192:6817/918056390] boot 2023-07-20T18:56:38.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:37 smithi057 ceph-mon[126340]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T18:56:38.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:37 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:38.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:37 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi192 2023-07-20T18:56:38.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:37 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi192 to 5722M 2023-07-20T18:56:38.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:37 smithi192 ceph-mon[129200]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:38.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:37 smithi192 ceph-mon[129200]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:56:38.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:37 smithi192 ceph-mon[129200]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T18:56:38.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:37 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:38.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:37 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:38.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:37 smithi192 ceph-mon[129200]: osd.8 [v2:172.21.15.192:6816/918056390,v1:172.21.15.192:6817/918056390] boot 2023-07-20T18:56:38.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:37 smithi192 ceph-mon[129200]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T18:56:38.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:37 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:38.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:37 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi192 2023-07-20T18:56:38.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:37 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi192 to 5722M 2023-07-20T18:56:38.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:37 smithi088 ceph-mon[128823]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T18:56:38.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:37 smithi088 ceph-mon[128823]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:56:38.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:37 smithi088 ceph-mon[128823]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T18:56:38.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:37 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:38.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:37 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:38.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:37 smithi088 ceph-mon[128823]: osd.8 [v2:172.21.15.192:6816/918056390,v1:172.21.15.192:6817/918056390] boot 2023-07-20T18:56:38.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:37 smithi088 ceph-mon[128823]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T18:56:38.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:37 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T18:56:38.982 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:38 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:56:38.983 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:38 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:56:38.983 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:38 smithi057 ceph-mon[126340]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T18:56:39.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:38 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:56:39.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:38 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:56:39.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:38 smithi192 ceph-mon[129200]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T18:56:39.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:38 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:56:39.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:38 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:56:39.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:38 smithi088 ceph-mon[128823]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T18:56:39.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.644+0000 7f32b5c99700 1 -- 172.21.15.57:0/3015427576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b0100800 msgr2=0x7f32b0100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:56:39.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.644+0000 7f32b5c99700 1 --2- 172.21.15.57:0/3015427576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b0100800 0x7f32b0100c20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f32a00099c0 tx=0x7f32a0009cd0 comp rx=0 tx=0).stop 2023-07-20T18:56:39.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.644+0000 7f32b5c99700 1 -- 172.21.15.57:0/3015427576 shutdown_connections 2023-07-20T18:56:39.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.644+0000 7f32b5c99700 1 --2- 172.21.15.57:0/3015427576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b01023a0 0x7f32b0108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:39.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.644+0000 7f32b5c99700 1 --2- 172.21.15.57:0/3015427576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b0100800 0x7f32b0100c20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:39.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.644+0000 7f32b5c99700 1 --2- 172.21.15.57:0/3015427576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32b0101a00 0x7f32b0101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:39.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.644+0000 7f32b5c99700 1 -- 172.21.15.57:0/3015427576 >> 172.21.15.57:0/3015427576 conn(0x7f32b00fbd60 msgr2=0x7f32b00fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:56:39.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.644+0000 7f32b5c99700 1 -- 172.21.15.57:0/3015427576 shutdown_connections 2023-07-20T18:56:39.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.644+0000 7f32b5c99700 1 -- 172.21.15.57:0/3015427576 wait complete. 2023-07-20T18:56:39.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.645+0000 7f32b5c99700 1 Processor -- start 2023-07-20T18:56:39.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.645+0000 7f32b5c99700 1 -- start start 2023-07-20T18:56:39.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32b5c99700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b0100800 0x7f32b01965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:56:39.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32b5c99700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32b0101a00 0x7f32b0196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:56:39.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32b5c99700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b01023a0 0x7f32b019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:56:39.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32b5c99700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b019e180 con 0x7f32b0100800 2023-07-20T18:56:39.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32b5c99700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b019e2c0 con 0x7f32b01023a0 2023-07-20T18:56:39.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32b5c99700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b019e450 con 0x7f32b0101a00 2023-07-20T18:56:39.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32af7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b0100800 0x7f32b01965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:56:39.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32af7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b0100800 0x7f32b01965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58866/0 (socket says 172.21.15.57:58866) 2023-07-20T18:56:39.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32af7fe700 1 -- 172.21.15.57:0/3978110553 learned_addr learned my addr 172.21.15.57:0/3978110553 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:56:39.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32aeffd700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32b0101a00 0x7f32b0196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:56:39.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32affff700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b01023a0 0x7f32b019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:56:39.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32af7fe700 1 -- 172.21.15.57:0/3978110553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32b0101a00 msgr2=0x7f32b0196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:56:39.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32af7fe700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32b0101a00 0x7f32b0196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:39.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32af7fe700 1 -- 172.21.15.57:0/3978110553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b01023a0 msgr2=0x7f32b019dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:56:39.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32af7fe700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b01023a0 0x7f32b019dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:56:39.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32af7fe700 1 -- 172.21.15.57:0/3978110553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a00095f0 con 0x7f32b0100800 2023-07-20T18:56:39.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.646+0000 7f32affff700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b01023a0 0x7f32b019dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T18:56:39.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.647+0000 7f32af7fe700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b0100800 0x7f32b01965f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009990 tx=0x7f32a000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:56:39.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.647+0000 7f32acff9700 1 -- 172.21.15.57:0/3978110553 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a001d070 con 0x7f32b0100800 2023-07-20T18:56:39.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.647+0000 7f32acff9700 1 -- 172.21.15.57:0/3978110553 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f32a0005030 con 0x7f32b0100800 2023-07-20T18:56:39.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.647+0000 7f32b5c99700 1 -- 172.21.15.57:0/3978110553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b019e6d0 con 0x7f32b0100800 2023-07-20T18:56:39.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.647+0000 7f32acff9700 1 -- 172.21.15.57:0/3978110553 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0004c40 con 0x7f32b0100800 2023-07-20T18:56:39.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.647+0000 7f32b5c99700 1 -- 172.21.15.57:0/3978110553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b019eae0 con 0x7f32b0100800 2023-07-20T18:56:39.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.648+0000 7f32b5c99700 1 -- 172.21.15.57:0/3978110553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b010c190 con 0x7f32b0100800 2023-07-20T18:56:39.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.650+0000 7f32acff9700 1 -- 172.21.15.57:0/3978110553 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f32a00051a0 con 0x7f32b0100800 2023-07-20T18:56:39.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.652+0000 7f32acff9700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f32980a1150 0x7f32980a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:56:39.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.653+0000 7f32acff9700 1 -- 172.21.15.57:0/3978110553 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f32a00cce60 con 0x7f32b0100800 2023-07-20T18:56:39.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.653+0000 7f32aeffd700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f32980a1150 0x7f32980a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:56:39.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.653+0000 7f32aeffd700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f32980a1150 0x7f32980a3610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f32a4005950 tx=0x7f32a400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:56:39.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.657+0000 7f32acff9700 1 -- 172.21.15.57:0/3978110553 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f32a0096f30 con 0x7f32b0100800 2023-07-20T18:56:39.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:56:39.874+0000 7f32b5c99700 1 -- 172.21.15.57:0/3978110553 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f32b0060bb0 con 0x7f32980a1150 2023-07-20T18:56:40.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:39 smithi057 ceph-mon[126340]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:40.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:39 smithi057 ceph-mon[126340]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T18:56:40.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:39 smithi192 ceph-mon[129200]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:40.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:39 smithi192 ceph-mon[129200]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T18:56:40.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:39 smithi088 ceph-mon[128823]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:40.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:39 smithi088 ceph-mon[128823]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T18:56:41.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:40 smithi057 ceph-mon[126340]: from='client.14343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:56:41.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:40 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:56:41.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:40 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:56:41.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:40 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:41.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:40 smithi192 ceph-mon[129200]: from='client.14343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:56:41.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:40 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:56:41.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:40 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:56:41.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:40 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:41.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:40 smithi088 ceph-mon[128823]: from='client.14343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:56:41.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:40 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:56:41.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:40 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:56:41.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:40 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:42.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:41 smithi057 ceph-mon[126340]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:42.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:41 smithi057 ceph-mon[126340]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T18:56:42.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:41 smithi192 ceph-mon[129200]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:42.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:41 smithi192 ceph-mon[129200]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T18:56:42.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:41 smithi088 ceph-mon[128823]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:42.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:41 smithi088 ceph-mon[128823]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T18:56:43.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:42 smithi057 ceph-mon[126340]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T18:56:43.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:42 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/986751559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d9cb69d-ed7b-4009-9ec5-027becc577d4"}]: dispatch 2023-07-20T18:56:43.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:42 smithi192 ceph-mon[129200]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T18:56:43.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:42 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/986751559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d9cb69d-ed7b-4009-9ec5-027becc577d4"}]: dispatch 2023-07-20T18:56:43.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:42 smithi088 ceph-mon[128823]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T18:56:43.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:42 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/986751559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d9cb69d-ed7b-4009-9ec5-027becc577d4"}]: dispatch 2023-07-20T18:56:44.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:43 smithi057 ceph-mon[126340]: pgmap v191: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T18:56:44.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:43 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/986751559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d9cb69d-ed7b-4009-9ec5-027becc577d4"}]': finished 2023-07-20T18:56:44.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:43 smithi057 ceph-mon[126340]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T18:56:44.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:56:44.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:43 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3080340071' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:56:44.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:43 smithi192 ceph-mon[129200]: pgmap v191: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T18:56:44.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:43 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/986751559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d9cb69d-ed7b-4009-9ec5-027becc577d4"}]': finished 2023-07-20T18:56:44.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:43 smithi192 ceph-mon[129200]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T18:56:44.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:56:44.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:43 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3080340071' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:56:44.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:43 smithi088 ceph-mon[128823]: pgmap v191: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T18:56:44.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:43 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/986751559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d9cb69d-ed7b-4009-9ec5-027becc577d4"}]': finished 2023-07-20T18:56:44.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:43 smithi088 ceph-mon[128823]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T18:56:44.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:56:44.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:43 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3080340071' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:56:46.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:45 smithi057 ceph-mon[126340]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 116 KiB/s, 0 objects/s recovering 2023-07-20T18:56:46.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:45 smithi192 ceph-mon[129200]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 116 KiB/s, 0 objects/s recovering 2023-07-20T18:56:46.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:45 smithi088 ceph-mon[128823]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 116 KiB/s, 0 objects/s recovering 2023-07-20T18:56:48.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:47 smithi057 ceph-mon[126340]: pgmap v194: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T18:56:48.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:47 smithi192 ceph-mon[129200]: pgmap v194: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T18:56:48.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:47 smithi088 ceph-mon[128823]: pgmap v194: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T18:56:50.232 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:49 smithi057 ceph-mon[126340]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-07-20T18:56:50.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:49 smithi192 ceph-mon[129200]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-07-20T18:56:50.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:49 smithi088 ceph-mon[128823]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-07-20T18:56:51.216 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T18:56:51.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:50 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:51.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T18:56:51.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:50 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:51.379 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:50 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T18:56:51.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:50 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:56:52.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:51 smithi057 ceph-mon[126340]: Deploying daemon osd.9 on smithi057 2023-07-20T18:56:52.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:51 smithi057 ceph-mon[126340]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:52.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:51 smithi192 ceph-mon[129200]: Deploying daemon osd.9 on smithi057 2023-07-20T18:56:52.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:51 smithi192 ceph-mon[129200]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:52.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:51 smithi088 ceph-mon[128823]: Deploying daemon osd.9 on smithi057 2023-07-20T18:56:52.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:51 smithi088 ceph-mon[128823]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:53.971 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:53 smithi057 ceph-mon[126340]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:54.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:53 smithi192 ceph-mon[129200]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:54.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:53 smithi088 ceph-mon[128823]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:56.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:55 smithi057 ceph-mon[126340]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:56.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:55 smithi192 ceph-mon[129200]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:56.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:55 smithi088 ceph-mon[128823]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:57.880 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:57.880 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:57.880 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:57 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:57.880 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:57 smithi088 ceph-mon[128823]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:57.888 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:57.888 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:57.888 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:57 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:57.888 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:57 smithi057 ceph-mon[126340]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:58.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:58.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:56:58.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:57 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:56:58.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:57 smithi192 ceph-mon[129200]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:56:59.983 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:56:59 smithi057 ceph-mon[126340]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:57:00.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.031+0000 7f32acff9700 1 -- 172.21.15.57:0/3978110553 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f32b0060bb0 con 0x7f32980a1150 2023-07-20T18:57:00.080 INFO:teuthology.orchestra.run.smithi057.stdout:Created osd(s) 9 on host 'smithi057' 2023-07-20T18:57:00.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.034+0000 7f32b5c99700 1 -- 172.21.15.57:0/3978110553 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f32980a1150 msgr2=0x7f32980a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:00.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.034+0000 7f32b5c99700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f32980a1150 0x7f32980a3610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f32a4005950 tx=0x7f32a400b410 comp rx=0 tx=0).stop 2023-07-20T18:57:00.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.034+0000 7f32b5c99700 1 -- 172.21.15.57:0/3978110553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b0100800 msgr2=0x7f32b01965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:00.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.034+0000 7f32b5c99700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b0100800 0x7f32b01965f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009990 tx=0x7f32a000f690 comp rx=0 tx=0).stop 2023-07-20T18:57:00.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.034+0000 7f32b5c99700 1 -- 172.21.15.57:0/3978110553 shutdown_connections 2023-07-20T18:57:00.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.034+0000 7f32b5c99700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f32980a1150 0x7f32980a3610 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:00.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.034+0000 7f32b5c99700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b01023a0 0x7f32b019dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:00.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.034+0000 7f32b5c99700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b0100800 0x7f32b01965f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:00.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.034+0000 7f32b5c99700 1 --2- 172.21.15.57:0/3978110553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32b0101a00 0x7f32b0196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:00.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.034+0000 7f32b5c99700 1 -- 172.21.15.57:0/3978110553 >> 172.21.15.57:0/3978110553 conn(0x7f32b00fbd60 msgr2=0x7f32b00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:00.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.035+0000 7f32b5c99700 1 -- 172.21.15.57:0/3978110553 shutdown_connections 2023-07-20T18:57:00.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:00.035+0000 7f32b5c99700 1 -- 172.21.15.57:0/3978110553 wait complete. 2023-07-20T18:57:00.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:56:59 smithi192 ceph-mon[129200]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:57:00.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:56:59 smithi088 ceph-mon[128823]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:57:00.502 DEBUG:teuthology.orchestra.run.smithi057:osd.9> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.9.service 2023-07-20T18:57:00.506 INFO:tasks.cephadm:Deploying osd.10 on smithi088 with /dev/vg_nvme/lv_1... 2023-07-20T18:57:00.507 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T18:57:00.587 INFO:journalctl@ceph.osd.9.smithi057.stdout:-- Logs begin at Thu 2023-07-20 18:30:45 UTC. -- 2023-07-20T18:57:00.983 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:57:00 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:57:00.574+0000 7efd7bd88640 -1 osd.9 0 log_to_monitors true 2023-07-20T18:57:01.074 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T18:57:01.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:01 smithi088 ceph-mon[128823]: from='osd.9 [v2:172.21.15.57:6826/1773340763,v1:172.21.15.57:6827/1773340763]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T18:57:01.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:01 smithi088 ceph-mon[128823]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T18:57:01.309 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:01 smithi088 ceph-mon[128823]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:57:01.337 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:01 smithi192 ceph-mon[129200]: from='osd.9 [v2:172.21.15.57:6826/1773340763,v1:172.21.15.57:6827/1773340763]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T18:57:01.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:01 smithi192 ceph-mon[129200]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T18:57:01.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:01 smithi192 ceph-mon[129200]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:57:01.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:01.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:01 smithi057 ceph-mon[126340]: from='osd.9 [v2:172.21.15.57:6826/1773340763,v1:172.21.15.57:6827/1773340763]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T18:57:01.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:01 smithi057 ceph-mon[126340]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T18:57:01.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:01 smithi057 ceph-mon[126340]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:57:02.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:02 smithi088 ceph-mon[128823]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T18:57:02.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:02 smithi088 ceph-mon[128823]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T18:57:02.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:02.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:02 smithi088 ceph-mon[128823]: from='osd.9 [v2:172.21.15.57:6826/1773340763,v1:172.21.15.57:6827/1773340763]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:57:02.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:02 smithi088 ceph-mon[128823]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:57:02.439 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T18:57:02.439 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T18:57:02.439 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-20T18:57:02.439 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-20T18:57:02.439 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107665 s, 974 MB/s 2023-07-20T18:57:02.440 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-20T18:57:02.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:02 smithi057 ceph-mon[126340]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T18:57:02.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:02 smithi057 ceph-mon[126340]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T18:57:02.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:02.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:02 smithi057 ceph-mon[126340]: from='osd.9 [v2:172.21.15.57:6826/1773340763,v1:172.21.15.57:6827/1773340763]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:57:02.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:02 smithi057 ceph-mon[126340]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:57:02.484 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:57:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:57:02.068+0000 7efd71c01700 -1 osd.9 0 waiting for initial osdmap 2023-07-20T18:57:02.484 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:57:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:57:02.088+0000 7efd6a1f0700 -1 osd.9 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:57:02.587 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:02 smithi192 ceph-mon[129200]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T18:57:02.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:02 smithi192 ceph-mon[129200]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T18:57:02.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:02.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:02 smithi192 ceph-mon[129200]: from='osd.9 [v2:172.21.15.57:6826/1773340763,v1:172.21.15.57:6827/1773340763]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:57:02.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:02 smithi192 ceph-mon[129200]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2023-07-20T18:57:02.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_1 2023-07-20T18:57:03.365 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T18:57:03.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:03 smithi088 ceph-mon[128823]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:57:03.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:03 smithi088 ceph-mon[128823]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T18:57:03.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:03.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:03.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:03 smithi088 ceph-mon[128823]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:57:03.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:03 smithi057 ceph-mon[126340]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:57:03.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:03 smithi057 ceph-mon[126340]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T18:57:03.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:03.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:03.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:03 smithi057 ceph-mon[126340]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:57:03.587 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:03 smithi192 ceph-mon[129200]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2023-07-20T18:57:03.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:03 smithi192 ceph-mon[129200]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T18:57:03.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:03.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:03.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:03 smithi192 ceph-mon[129200]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T18:57:04.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:57:04.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:57:04.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:04.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: osd.9 [v2:172.21.15.57:6826/1773340763,v1:172.21.15.57:6827/1773340763] boot 2023-07-20T18:57:04.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T18:57:04.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:04.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi057 2023-07-20T18:57:04.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi057 to 4318M 2023-07-20T18:57:04.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:04.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:04.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.373+0000 7efdea498700 1 -- 172.21.15.88:0/1569368687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efde4100900 msgr2=0x7efde4100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.373+0000 7efdea498700 1 --2- 172.21.15.88:0/1569368687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efde4100900 0x7efde4100d60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efdd40099c0 tx=0x7efdd4009cd0 comp rx=0 tx=0).stop 2023-07-20T18:57:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.373+0000 7efdea498700 1 -- 172.21.15.88:0/1569368687 shutdown_connections 2023-07-20T18:57:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.373+0000 7efdea498700 1 --2- 172.21.15.88:0/1569368687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efde41012a0 0x7efde4107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.373+0000 7efdea498700 1 --2- 172.21.15.88:0/1569368687 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efde40ff700 0x7efde40ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.373+0000 7efdea498700 1 --2- 172.21.15.88:0/1569368687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efde4100900 0x7efde4100d60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.373+0000 7efdea498700 1 -- 172.21.15.88:0/1569368687 >> 172.21.15.88:0/1569368687 conn(0x7efde40fac80 msgr2=0x7efde40fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.373+0000 7efdea498700 1 -- 172.21.15.88:0/1569368687 shutdown_connections 2023-07-20T18:57:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.373+0000 7efdea498700 1 -- 172.21.15.88:0/1569368687 wait complete. 2023-07-20T18:57:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.373+0000 7efdea498700 1 Processor -- start 2023-07-20T18:57:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efdea498700 1 -- start start 2023-07-20T18:57:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efdea498700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efde40ff700 0x7efde4069640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efdea498700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efde4100900 0x7efde419b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efdea498700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efde41012a0 0x7efde419d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efdea498700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde419e180 con 0x7efde41012a0 2023-07-20T18:57:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efdea498700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde419e2f0 con 0x7efde40ff700 2023-07-20T18:57:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efdea498700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde419e4b0 con 0x7efde4100900 2023-07-20T18:57:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efde40ff700 0x7efde4069640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efde40ff700 0x7efde4069640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46250/0 (socket says 172.21.15.88:46250) 2023-07-20T18:57:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde3fff700 1 -- 172.21.15.88:0/1728631452 learned_addr learned my addr 172.21.15.88:0/1728631452 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-20T18:57:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde37fe700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efde4100900 0x7efde419b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde3fff700 1 -- 172.21.15.88:0/1728631452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efde4100900 msgr2=0x7efde419b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde3fff700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efde4100900 0x7efde419b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde8a35700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efde41012a0 0x7efde419d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde3fff700 1 -- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efde41012a0 msgr2=0x7efde419d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde3fff700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efde41012a0 0x7efde419d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde3fff700 1 -- 172.21.15.88:0/1728631452 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd40095f0 con 0x7efde40ff700 2023-07-20T18:57:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde8a35700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efde41012a0 0x7efde419d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T18:57:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.374+0000 7efde3fff700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efde40ff700 0x7efde4069640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efdd800b500 tx=0x7efdd800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.375+0000 7efde17fa700 1 -- 172.21.15.88:0/1728631452 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8011830 con 0x7efde40ff700 2023-07-20T18:57:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.375+0000 7efdea498700 1 -- 172.21.15.88:0/1728631452 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde41951c0 con 0x7efde40ff700 2023-07-20T18:57:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.375+0000 7efde17fa700 1 -- 172.21.15.88:0/1728631452 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7efdd8011e70 con 0x7efde40ff700 2023-07-20T18:57:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.375+0000 7efde17fa700 1 -- 172.21.15.88:0/1728631452 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd800f2e0 con 0x7efde40ff700 2023-07-20T18:57:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.375+0000 7efdea498700 1 -- 172.21.15.88:0/1728631452 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde41956c0 con 0x7efde40ff700 2023-07-20T18:57:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.376+0000 7efdea498700 1 -- 172.21.15.88:0/1728631452 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde410b0e0 con 0x7efde40ff700 2023-07-20T18:57:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.377+0000 7efde17fa700 1 -- 172.21.15.88:0/1728631452 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efdd80105f0 con 0x7efde40ff700 2023-07-20T18:57:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.378+0000 7efde17fa700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efdcc0a1150 0x7efdcc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.378+0000 7efde17fa700 1 -- 172.21.15.88:0/1728631452 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6261+0+0 (secure 0 0 0) 0x7efdd80cc180 con 0x7efde40ff700 2023-07-20T18:57:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.378+0000 7efde37fe700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efdcc0a1150 0x7efdcc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.379+0000 7efde37fe700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efdcc0a1150 0x7efdcc0a3610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efdd4006010 tx=0x7efdd4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.381+0000 7efde17fa700 1 -- 172.21.15.88:0/1728631452 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efdd8096030 con 0x7efde40ff700 2023-07-20T18:57:04.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:57:04.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:57:04.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:04.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: osd.9 [v2:172.21.15.57:6826/1773340763,v1:172.21.15.57:6827/1773340763] boot 2023-07-20T18:57:04.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T18:57:04.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:04.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi057 2023-07-20T18:57:04.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi057 to 4318M 2023-07-20T18:57:04.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:04.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:04.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:04.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:04.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:04.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:57:04.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:57:04.588 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:04.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: osd.9 [v2:172.21.15.57:6826/1773340763,v1:172.21.15.57:6827/1773340763] boot 2023-07-20T18:57:04.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T18:57:04.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T18:57:04.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi057 2023-07-20T18:57:04.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:04.591 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi057 to 4318M 2023-07-20T18:57:04.591 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.591 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:04.591 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:04.592 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.592 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:04.592 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:04.593 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:04.593 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:04.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:04.576+0000 7efdea498700 1 -- 172.21.15.88:0/1728631452 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7efde4196260 con 0x7efdcc0a1150 2023-07-20T18:57:05.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:05 smithi088 ceph-mon[128823]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T18:57:05.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:05 smithi088 ceph-mon[128823]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:57:05.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:05 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:57:05.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:05 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:57:05.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:05 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:05.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:05 smithi088 ceph-mon[128823]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:05.838 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:05 smithi192 ceph-mon[129200]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T18:57:05.838 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:05 smithi192 ceph-mon[129200]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:57:05.838 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:05 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:57:05.838 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:05 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:57:05.838 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:05 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:05.838 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:05 smithi192 ceph-mon[129200]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:05.983 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:05 smithi057 ceph-mon[126340]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T18:57:05.983 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:05 smithi057 ceph-mon[126340]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:57:05.983 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:05 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:57:05.983 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:05 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:57:05.984 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:05 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:05.984 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:05 smithi057 ceph-mon[126340]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:08.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:07 smithi192 ceph-mon[129200]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:08.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:07 smithi192 ceph-mon[129200]: from='client.? 172.21.15.88:0/858875109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5799219-0979-40b1-a596-acd37f37a72d"}]: dispatch 2023-07-20T18:57:08.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:07 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5799219-0979-40b1-a596-acd37f37a72d"}]: dispatch 2023-07-20T18:57:08.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:07 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5799219-0979-40b1-a596-acd37f37a72d"}]': finished 2023-07-20T18:57:08.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:07 smithi192 ceph-mon[129200]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T18:57:08.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:08.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:07 smithi192 ceph-mon[129200]: from='client.? 172.21.15.88:0/1022249144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:57:08.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:07 smithi088 ceph-mon[128823]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:08.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:07 smithi088 ceph-mon[128823]: from='client.? 172.21.15.88:0/858875109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5799219-0979-40b1-a596-acd37f37a72d"}]: dispatch 2023-07-20T18:57:08.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:07 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5799219-0979-40b1-a596-acd37f37a72d"}]: dispatch 2023-07-20T18:57:08.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:07 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5799219-0979-40b1-a596-acd37f37a72d"}]': finished 2023-07-20T18:57:08.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:07 smithi088 ceph-mon[128823]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T18:57:08.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:08.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:07 smithi088 ceph-mon[128823]: from='client.? 172.21.15.88:0/1022249144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:57:08.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:07 smithi057 ceph-mon[126340]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:08.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:07 smithi057 ceph-mon[126340]: from='client.? 172.21.15.88:0/858875109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5799219-0979-40b1-a596-acd37f37a72d"}]: dispatch 2023-07-20T18:57:08.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:07 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5799219-0979-40b1-a596-acd37f37a72d"}]: dispatch 2023-07-20T18:57:08.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:07 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5799219-0979-40b1-a596-acd37f37a72d"}]': finished 2023-07-20T18:57:08.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:07 smithi057 ceph-mon[126340]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T18:57:08.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:08.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:07 smithi057 ceph-mon[126340]: from='client.? 172.21.15.88:0/1022249144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:57:10.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:09 smithi192 ceph-mon[129200]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:10.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:09 smithi088 ceph-mon[128823]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:10.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:09 smithi057 ceph-mon[126340]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:12.062 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:11 smithi088 ceph-mon[128823]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:12.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:11 smithi192 ceph-mon[129200]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:12.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:11 smithi057 ceph-mon[126340]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:14.064 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:13 smithi088 ceph-mon[128823]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:14.065 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T18:57:14.065 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:14.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:13 smithi192 ceph-mon[129200]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:14.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T18:57:14.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:14.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:13 smithi057 ceph-mon[126340]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:14.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T18:57:14.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:14.880 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:14 smithi088 ceph-mon[128823]: Deploying daemon osd.10 on smithi088 2023-07-20T18:57:15.087 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:14 smithi192 ceph-mon[129200]: Deploying daemon osd.10 on smithi088 2023-07-20T18:57:15.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:14 smithi057 ceph-mon[126340]: Deploying daemon osd.10 on smithi088 2023-07-20T18:57:15.881 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:15 smithi088 ceph-mon[128823]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:16.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:15 smithi192 ceph-mon[129200]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:16.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:15 smithi057 ceph-mon[126340]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:18.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:17 smithi192 ceph-mon[129200]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:18.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:18.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:18.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:18.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:17 smithi088 ceph-mon[128823]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:18.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:18.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:18.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:18.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:17 smithi057 ceph-mon[126340]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:18.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:18.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:18.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:19.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:19 smithi088 ceph-mon[128823]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:20.085 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 10 on host 'smithi088' 2023-07-20T18:57:20.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.036+0000 7efde17fa700 1 -- 172.21.15.88:0/1728631452 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7efde4196260 con 0x7efdcc0a1150 2023-07-20T18:57:20.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.038+0000 7efdea498700 1 -- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efdcc0a1150 msgr2=0x7efdcc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:20.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efdcc0a1150 0x7efdcc0a3610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efdd4006010 tx=0x7efdd4000bc0 comp rx=0 tx=0).stop 2023-07-20T18:57:20.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 -- 172.21.15.88:0/1728631452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efde40ff700 msgr2=0x7efde4069640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:20.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efde40ff700 0x7efde4069640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efdd800b500 tx=0x7efdd800b8c0 comp rx=0 tx=0).stop 2023-07-20T18:57:20.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 -- 172.21.15.88:0/1728631452 shutdown_connections 2023-07-20T18:57:20.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7efdcc0a1150 0x7efdcc0a3610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:20.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efde41012a0 0x7efde419d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:20.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efde40ff700 0x7efde4069640 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:20.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 --2- 172.21.15.88:0/1728631452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efde4100900 0x7efde419b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:20.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 -- 172.21.15.88:0/1728631452 >> 172.21.15.88:0/1728631452 conn(0x7efde40fac80 msgr2=0x7efde40fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:20.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 -- 172.21.15.88:0/1728631452 shutdown_connections 2023-07-20T18:57:20.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-20T18:57:20.039+0000 7efdea498700 1 -- 172.21.15.88:0/1728631452 wait complete. 2023-07-20T18:57:20.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:19 smithi192 ceph-mon[129200]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:20.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:19 smithi057 ceph-mon[126340]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:20.544 DEBUG:teuthology.orchestra.run.smithi088:osd.10> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.10.service 2023-07-20T18:57:20.546 INFO:tasks.cephadm:Deploying osd.11 on smithi192 with /dev/vg_nvme/lv_1... 2023-07-20T18:57:20.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T18:57:20.582 INFO:journalctl@ceph.osd.10.smithi088.stdout:-- Logs begin at Thu 2023-07-20 18:25:07 UTC. -- 2023-07-20T18:57:21.109 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:57:21.201 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.201 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.201 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:21.201 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:21.202 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.202 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.202 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.202 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:20 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:21.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:21.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:21.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:20 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:21.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:21.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:21.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:21.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:20 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:21.518 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:57:21 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:57:21.318+0000 7fa16d4ae640 -1 osd.10 0 log_to_monitors true 2023-07-20T18:57:22.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:21 smithi088 ceph-mon[128823]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:22.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:21 smithi088 ceph-mon[128823]: from='osd.10 [v2:172.21.15.88:6824/2781975579,v1:172.21.15.88:6825/2781975579]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T18:57:22.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:21 smithi088 ceph-mon[128823]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T18:57:22.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:21 smithi057 ceph-mon[126340]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:22.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:21 smithi057 ceph-mon[126340]: from='osd.10 [v2:172.21.15.88:6824/2781975579,v1:172.21.15.88:6825/2781975579]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T18:57:22.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:21 smithi057 ceph-mon[126340]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T18:57:22.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:21 smithi192 ceph-mon[129200]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:22.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:21 smithi192 ceph-mon[129200]: from='osd.10 [v2:172.21.15.88:6824/2781975579,v1:172.21.15.88:6825/2781975579]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T18:57:22.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:21 smithi192 ceph-mon[129200]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T18:57:22.509 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T18:57:22.509 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T18:57:22.510 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-07-20T18:57:22.510 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-07-20T18:57:22.510 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125037 s, 839 MB/s 2023-07-20T18:57:22.511 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-07-20T18:57:23.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2023-07-20T18:57:23.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T18:57:23.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T18:57:23.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='osd.10 [v2:172.21.15.88:6824/2781975579,v1:172.21.15.88:6825/2781975579]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:57:23.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:23.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:57:23.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:23.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi088 2023-07-20T18:57:23.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi088 to 4292M 2023-07-20T18:57:23.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:23.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:23.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.381 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:57:23.029+0000 7fa162b26700 -1 osd.10 0 waiting for initial osdmap 2023-07-20T18:57:23.381 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:57:23.040+0000 7fa15c918700 -1 osd.10 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:57:23.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T18:57:23.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T18:57:23.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='osd.10 [v2:172.21.15.88:6824/2781975579,v1:172.21.15.88:6825/2781975579]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:57:23.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:23.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:57:23.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:23.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi088 2023-07-20T18:57:23.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi088 to 4292M 2023-07-20T18:57:23.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:23.387 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:23.387 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T18:57:23.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T18:57:23.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='osd.10 [v2:172.21.15.88:6824/2781975579,v1:172.21.15.88:6825/2781975579]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:57:23.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:23.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-20T18:57:23.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T18:57:23.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi088 2023-07-20T18:57:23.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:23.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi088 to 4292M 2023-07-20T18:57:23.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:23.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:23.488 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:23.619 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:57:24.132 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:24 smithi192 ceph-mon[129200]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:57:24.133 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:24 smithi192 ceph-mon[129200]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T18:57:24.133 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:24 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:24.133 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:24 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:24.133 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:24 smithi192 ceph-mon[129200]: osd.10 [v2:172.21.15.88:6824/2781975579,v1:172.21.15.88:6825/2781975579] boot 2023-07-20T18:57:24.133 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:24 smithi192 ceph-mon[129200]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T18:57:24.133 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:24 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:24.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:24 smithi088 ceph-mon[128823]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:57:24.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:24 smithi088 ceph-mon[128823]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T18:57:24.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:24 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:24.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:24 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:24.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:24 smithi088 ceph-mon[128823]: osd.10 [v2:172.21.15.88:6824/2781975579,v1:172.21.15.88:6825/2781975579] boot 2023-07-20T18:57:24.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:24 smithi088 ceph-mon[128823]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T18:57:24.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:24 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:24.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:24 smithi057 ceph-mon[126340]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-20T18:57:24.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:24 smithi057 ceph-mon[126340]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T18:57:24.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:24 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:24.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:24 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:24.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:24 smithi057 ceph-mon[126340]: osd.10 [v2:172.21.15.88:6824/2781975579,v1:172.21.15.88:6825/2781975579] boot 2023-07-20T18:57:24.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:24 smithi057 ceph-mon[126340]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T18:57:24.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:24 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T18:57:24.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.768+0000 7ff39d0ea700 1 -- 172.21.15.192:0/3326209171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3981023d0 msgr2=0x7ff398108c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:24.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.768+0000 7ff39d0ea700 1 --2- 172.21.15.192:0/3326209171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3981023d0 0x7ff398108c80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff39000b0a0 tx=0x7ff39000b3b0 comp rx=0 tx=0).stop 2023-07-20T18:57:24.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.768+0000 7ff39d0ea700 1 -- 172.21.15.192:0/3326209171 shutdown_connections 2023-07-20T18:57:24.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.768+0000 7ff39d0ea700 1 --2- 172.21.15.192:0/3326209171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3981023d0 0x7ff398108c80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:24.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.768+0000 7ff39d0ea700 1 --2- 172.21.15.192:0/3326209171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff398100830 0x7ff398100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:24.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.768+0000 7ff39d0ea700 1 --2- 172.21.15.192:0/3326209171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff398101a30 0x7ff398101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:24.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.768+0000 7ff39d0ea700 1 -- 172.21.15.192:0/3326209171 >> 172.21.15.192:0/3326209171 conn(0x7ff3980fbd90 msgr2=0x7ff3980fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:24.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.768+0000 7ff39d0ea700 1 -- 172.21.15.192:0/3326209171 shutdown_connections 2023-07-20T18:57:24.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.768+0000 7ff39d0ea700 1 -- 172.21.15.192:0/3326209171 wait complete. 2023-07-20T18:57:24.820 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39d0ea700 1 Processor -- start 2023-07-20T18:57:24.820 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39d0ea700 1 -- start start 2023-07-20T18:57:24.820 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39d0ea700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff398100830 0x7ff398074510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:24.820 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39d0ea700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff398101a30 0x7ff398072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:24.821 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39d0ea700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3981023d0 0x7ff3980730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:24.821 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39d0ea700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff398073820 con 0x7ff398100830 2023-07-20T18:57:24.821 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39d0ea700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff398073990 con 0x7ff398101a30 2023-07-20T18:57:24.821 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39d0ea700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3981a0780 con 0x7ff3981023d0 2023-07-20T18:57:24.822 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39759e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3981023d0 0x7ff3980730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:24.822 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39759e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3981023d0 0x7ff3980730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:48482/0 (socket says 172.21.15.192:48482) 2023-07-20T18:57:24.822 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39759e700 1 -- 172.21.15.192:0/1734946546 learned_addr learned my addr 172.21.15.192:0/1734946546 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-07-20T18:57:24.822 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39659c700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff398101a30 0x7ff398072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:24.822 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39759e700 1 -- 172.21.15.192:0/1734946546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff398101a30 msgr2=0x7ff398072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:24.822 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39759e700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff398101a30 0x7ff398072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:24.823 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39759e700 1 -- 172.21.15.192:0/1734946546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff398100830 msgr2=0x7ff398074510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:24.823 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39759e700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff398100830 0x7ff398074510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:24.823 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.769+0000 7ff39759e700 1 -- 172.21.15.192:0/1734946546 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff390009d00 con 0x7ff3981023d0 2023-07-20T18:57:24.823 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.770+0000 7ff39759e700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3981023d0 0x7ff3980730a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff3900062a0 tx=0x7ff390009570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:24.823 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.770+0000 7ff37ffff700 1 -- 172.21.15.192:0/1734946546 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39000e040 con 0x7ff3981023d0 2023-07-20T18:57:24.823 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.770+0000 7ff39d0ea700 1 -- 172.21.15.192:0/1734946546 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3981a0a00 con 0x7ff3981023d0 2023-07-20T18:57:24.824 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.770+0000 7ff37ffff700 1 -- 172.21.15.192:0/1734946546 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7ff39001db60 con 0x7ff3981023d0 2023-07-20T18:57:24.824 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.770+0000 7ff37ffff700 1 -- 172.21.15.192:0/1734946546 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39001bd30 con 0x7ff3981023d0 2023-07-20T18:57:24.824 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.770+0000 7ff39d0ea700 1 -- 172.21.15.192:0/1734946546 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3981a0f40 con 0x7ff3981023d0 2023-07-20T18:57:24.824 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.771+0000 7ff39d0ea700 1 -- 172.21.15.192:0/1734946546 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff398107d80 con 0x7ff3981023d0 2023-07-20T18:57:24.825 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.772+0000 7ff37ffff700 1 -- 172.21.15.192:0/1734946546 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff39001dcd0 con 0x7ff3981023d0 2023-07-20T18:57:24.825 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.772+0000 7ff37ffff700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff3800a1150 0x7ff3800a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:24.825 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.772+0000 7ff37ffff700 1 -- 172.21.15.192:0/1734946546 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6723+0+0 (secure 0 0 0) 0x7ff3900ce430 con 0x7ff3981023d0 2023-07-20T18:57:24.826 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.773+0000 7ff396d9d700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff3800a1150 0x7ff3800a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:24.826 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.773+0000 7ff396d9d700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff3800a1150 0x7ff3800a3610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff398101890 tx=0x7ff38c009470 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:24.826 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.776+0000 7ff37ffff700 1 -- 172.21.15.192:0/1734946546 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff390098240 con 0x7ff3981023d0 2023-07-20T18:57:25.018 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:24.969+0000 7ff39d0ea700 1 -- 172.21.15.192:0/1734946546 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff398060bb0 con 0x7ff3800a1150 2023-07-20T18:57:25.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:25 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:57:25.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:25 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:57:25.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:25 smithi192 ceph-mon[129200]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:25.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:25 smithi192 ceph-mon[129200]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T18:57:25.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:25 smithi192 ceph-mon[129200]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:57:25.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:57:25.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:57:25.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:25.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:25 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:57:25.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:25 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:57:25.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:25 smithi088 ceph-mon[128823]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:25.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:25 smithi088 ceph-mon[128823]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T18:57:25.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:25 smithi088 ceph-mon[128823]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:57:25.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:57:25.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:57:25.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:25.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:25 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:57:25.483 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:25 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:57:25.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:25 smithi057 ceph-mon[126340]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:25.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:25 smithi057 ceph-mon[126340]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T18:57:25.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:25 smithi057 ceph-mon[126340]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:57:25.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T18:57:25.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T18:57:25.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:27.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:26 smithi192 ceph-mon[129200]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T18:57:27.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:26 smithi088 ceph-mon[128823]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T18:57:27.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:26 smithi057 ceph-mon[126340]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T18:57:28.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:27 smithi192 ceph-mon[129200]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:28.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:27 smithi192 ceph-mon[129200]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T18:57:28.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:27 smithi192 ceph-mon[129200]: from='client.? 172.21.15.192:0/1142595884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e77cac6c-107e-41f9-8efd-e1140c4e3bf7"}]: dispatch 2023-07-20T18:57:28.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:27 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e77cac6c-107e-41f9-8efd-e1140c4e3bf7"}]: dispatch 2023-07-20T18:57:28.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:27 smithi192 ceph-mon[129200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e77cac6c-107e-41f9-8efd-e1140c4e3bf7"}]': finished 2023-07-20T18:57:28.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:27 smithi192 ceph-mon[129200]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T18:57:28.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:27 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:27 smithi088 ceph-mon[128823]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:27 smithi088 ceph-mon[128823]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T18:57:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:27 smithi088 ceph-mon[128823]: from='client.? 172.21.15.192:0/1142595884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e77cac6c-107e-41f9-8efd-e1140c4e3bf7"}]: dispatch 2023-07-20T18:57:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:27 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e77cac6c-107e-41f9-8efd-e1140c4e3bf7"}]: dispatch 2023-07-20T18:57:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:27 smithi088 ceph-mon[128823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e77cac6c-107e-41f9-8efd-e1140c4e3bf7"}]': finished 2023-07-20T18:57:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:27 smithi088 ceph-mon[128823]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T18:57:28.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:27 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:28.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:27 smithi057 ceph-mon[126340]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:28.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:27 smithi057 ceph-mon[126340]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T18:57:28.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:27 smithi057 ceph-mon[126340]: from='client.? 172.21.15.192:0/1142595884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e77cac6c-107e-41f9-8efd-e1140c4e3bf7"}]: dispatch 2023-07-20T18:57:28.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:27 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e77cac6c-107e-41f9-8efd-e1140c4e3bf7"}]: dispatch 2023-07-20T18:57:28.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:27 smithi057 ceph-mon[126340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e77cac6c-107e-41f9-8efd-e1140c4e3bf7"}]': finished 2023-07-20T18:57:28.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:27 smithi057 ceph-mon[126340]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T18:57:28.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:29.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:28 smithi192 ceph-mon[129200]: from='client.? 172.21.15.192:0/615970168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:57:29.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:28 smithi088 ceph-mon[128823]: from='client.? 172.21.15.192:0/615970168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:57:29.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:28 smithi057 ceph-mon[126340]: from='client.? 172.21.15.192:0/615970168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T18:57:30.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:29 smithi192 ceph-mon[129200]: pgmap v227: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T18:57:30.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:29 smithi088 ceph-mon[128823]: pgmap v227: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T18:57:30.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:29 smithi057 ceph-mon[126340]: pgmap v227: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T18:57:32.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:31 smithi192 ceph-mon[129200]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T18:57:32.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:31 smithi088 ceph-mon[128823]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T18:57:32.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:31 smithi057 ceph-mon[126340]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T18:57:33.836 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:33 smithi192 ceph-mon[129200]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-07-20T18:57:34.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:33 smithi088 ceph-mon[128823]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-07-20T18:57:34.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:33 smithi057 ceph-mon[126340]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-07-20T18:57:35.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:34 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T18:57:35.088 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:34 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:35.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:34 smithi192 ceph-mon[129200]: Deploying daemon osd.11 on smithi192 2023-07-20T18:57:35.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:34 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T18:57:35.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:34 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:35.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:34 smithi088 ceph-mon[128823]: Deploying daemon osd.11 on smithi192 2023-07-20T18:57:35.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:34 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T18:57:35.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:34 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:35.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:34 smithi057 ceph-mon[126340]: Deploying daemon osd.11 on smithi192 2023-07-20T18:57:36.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:35 smithi192 ceph-mon[129200]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T18:57:36.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:35 smithi088 ceph-mon[128823]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T18:57:36.233 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:35 smithi057 ceph-mon[126340]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T18:57:38.069 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:37 smithi192 ceph-mon[129200]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T18:57:38.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:37 smithi088 ceph-mon[128823]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T18:57:38.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:37 smithi057 ceph-mon[126340]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T18:57:39.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:39 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:39.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:39 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:39.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:39 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:39.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:39 smithi192 ceph-mon[129200]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:39.734 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:39 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:39.734 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:39 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:39.734 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:39 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:39.734 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:39 smithi057 ceph-mon[126340]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:39.881 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:39 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:39.881 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:39 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:39.881 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:39 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:39.881 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:39 smithi088 ceph-mon[128823]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:41.184 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 11 on host 'smithi192' 2023-07-20T18:57:41.184 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.136+0000 7ff37ffff700 1 -- 172.21.15.192:0/1734946546 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7ff398060bb0 con 0x7ff3800a1150 2023-07-20T18:57:41.186 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 -- 172.21.15.192:0/1734946546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff3800a1150 msgr2=0x7ff3800a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:41.186 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff3800a1150 0x7ff3800a3610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff398101890 tx=0x7ff38c009470 comp rx=0 tx=0).stop 2023-07-20T18:57:41.186 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 -- 172.21.15.192:0/1734946546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3981023d0 msgr2=0x7ff3980730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:41.186 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3981023d0 0x7ff3980730a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff3900062a0 tx=0x7ff390009570 comp rx=0 tx=0).stop 2023-07-20T18:57:41.187 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 -- 172.21.15.192:0/1734946546 shutdown_connections 2023-07-20T18:57:41.187 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff3800a1150 0x7ff3800a3610 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:41.187 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3981023d0 0x7ff3980730a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:41.187 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff398100830 0x7ff398074510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:41.187 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 --2- 172.21.15.192:0/1734946546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff398101a30 0x7ff398072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:41.187 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 -- 172.21.15.192:0/1734946546 >> 172.21.15.192:0/1734946546 conn(0x7ff3980fbd90 msgr2=0x7ff398104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:41.188 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 -- 172.21.15.192:0/1734946546 shutdown_connections 2023-07-20T18:57:41.188 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:41.138+0000 7ff39d0ea700 1 -- 172.21.15.192:0/1734946546 wait complete. 2023-07-20T18:57:41.608 DEBUG:teuthology.orchestra.run.smithi192:osd.11> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.11.service 2023-07-20T18:57:41.610 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-20T18:57:41.610 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T18:57:41.639 INFO:journalctl@ceph.osd.11.smithi192.stdout:-- Logs begin at Thu 2023-07-20 18:24:33 UTC. -- 2023-07-20T18:57:41.984 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:41 smithi057 ceph-mon[126340]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:41.984 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:41.984 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:41.984 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:41.984 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:41.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:41.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:41.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:41.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:41 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:42.111 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:41 smithi192 ceph-mon[129200]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:42.112 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:42.112 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:42.112 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:42.113 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:42.113 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:42.113 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:42.114 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:42.114 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:41 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:42.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:41 smithi088 ceph-mon[128823]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:42.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:42.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:42.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:42.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:42.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:42.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:42.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:42.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:41 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:42.349 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:57:42.413 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:57:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:57:42.250+0000 7f74cc5ca640 -1 osd.11 0 log_to_monitors true 2023-07-20T18:57:43.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:42 smithi088 ceph-mon[128823]: from='osd.11 [v2:172.21.15.192:6824/2864190031,v1:172.21.15.192:6825/2864190031]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T18:57:43.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:42 smithi088 ceph-mon[128823]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T18:57:43.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:42 smithi057 ceph-mon[126340]: from='osd.11 [v2:172.21.15.192:6824/2864190031,v1:172.21.15.192:6825/2864190031]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T18:57:43.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:42 smithi057 ceph-mon[126340]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T18:57:43.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:42 smithi192 ceph-mon[129200]: from='osd.11 [v2:172.21.15.192:6824/2864190031,v1:172.21.15.192:6825/2864190031]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T18:57:43.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:42 smithi192 ceph-mon[129200]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T18:57:43.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.716+0000 7f03d830e700 1 -- 172.21.15.57:0/1936576760 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d0101310 msgr2=0x7f03d0103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:43.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.716+0000 7f03d830e700 1 --2- 172.21.15.57:0/1936576760 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d0101310 0x7f03d0103700 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f03cc009990 tx=0x7f03cc009ca0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.716+0000 7f03d830e700 1 -- 172.21.15.57:0/1936576760 shutdown_connections 2023-07-20T18:57:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.716+0000 7f03d830e700 1 --2- 172.21.15.57:0/1936576760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d0106570 0x7f03d0108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.716+0000 7f03d830e700 1 --2- 172.21.15.57:0/1936576760 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d0101310 0x7f03d0103700 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.716+0000 7f03d830e700 1 --2- 172.21.15.57:0/1936576760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d0103c40 0x7f03d0106030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.716+0000 7f03d830e700 1 -- 172.21.15.57:0/1936576760 >> 172.21.15.57:0/1936576760 conn(0x7f03d00fac80 msgr2=0x7f03d00fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.716+0000 7f03d830e700 1 -- 172.21.15.57:0/1936576760 shutdown_connections 2023-07-20T18:57:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.716+0000 7f03d830e700 1 -- 172.21.15.57:0/1936576760 wait complete. 2023-07-20T18:57:43.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.717+0000 7f03d830e700 1 Processor -- start 2023-07-20T18:57:43.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.717+0000 7f03d830e700 1 -- start start 2023-07-20T18:57:43.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.717+0000 7f03d830e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d0101310 0x7f03d0100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:43.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.717+0000 7f03d830e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d0103c40 0x7f03d00ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:43.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.717+0000 7f03d830e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d0106570 0x7f03d00ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:43.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.717+0000 7f03d830e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00fff80 con 0x7f03d0106570 2023-07-20T18:57:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.717+0000 7f03d830e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01000f0 con 0x7f03d0103c40 2023-07-20T18:57:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.717+0000 7f03d830e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01a05f0 con 0x7f03d0101310 2023-07-20T18:57:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d60aa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d0101310 0x7f03d0100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d60aa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d0101310 0x7f03d0100c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:53494/0 (socket says 172.21.15.57:53494) 2023-07-20T18:57:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d60aa700 1 -- 172.21.15.57:0/3067460435 learned_addr learned my addr 172.21.15.57:0/3067460435 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:57:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d68ab700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d0106570 0x7f03d00ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d58a9700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d0103c40 0x7f03d00ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d60aa700 1 -- 172.21.15.57:0/3067460435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d0103c40 msgr2=0x7f03d00ff2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:43.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d60aa700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d0103c40 0x7f03d00ff2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d60aa700 1 -- 172.21.15.57:0/3067460435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d0106570 msgr2=0x7f03d00ff800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:43.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d60aa700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d0106570 0x7f03d00ff800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d60aa700 1 -- 172.21.15.57:0/3067460435 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c0009520 con 0x7f03d0101310 2023-07-20T18:57:43.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.718+0000 7f03d60aa700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d0101310 0x7f03d0100c70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f03cc000c00 tx=0x7f03cc0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:43.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.719+0000 7f03c77fe700 1 -- 172.21.15.57:0/3067460435 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc01d070 con 0x7f03d0101310 2023-07-20T18:57:43.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.719+0000 7f03c77fe700 1 -- 172.21.15.57:0/3067460435 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f03cc00fbd0 con 0x7f03d0101310 2023-07-20T18:57:43.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.719+0000 7f03d830e700 1 -- 172.21.15.57:0/3067460435 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc0095f0 con 0x7f03d0101310 2023-07-20T18:57:43.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.719+0000 7f03c77fe700 1 -- 172.21.15.57:0/3067460435 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc017650 con 0x7f03d0101310 2023-07-20T18:57:43.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.719+0000 7f03d830e700 1 -- 172.21.15.57:0/3067460435 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d01a0b80 con 0x7f03d0101310 2023-07-20T18:57:43.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.720+0000 7f03d830e700 1 -- 172.21.15.57:0/3067460435 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d00fc800 con 0x7f03d0101310 2023-07-20T18:57:43.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.722+0000 7f03c77fe700 1 -- 172.21.15.57:0/3067460435 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f03cc021410 con 0x7f03d0101310 2023-07-20T18:57:43.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.723+0000 7f03c77fe700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f03bc0a1150 0x7f03bc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:43.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.724+0000 7f03c77fe700 1 -- 172.21.15.57:0/3067460435 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f03cc0cdfa0 con 0x7f03d0101310 2023-07-20T18:57:43.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.724+0000 7f03d58a9700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f03bc0a1150 0x7f03bc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:43.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.724+0000 7f03d58a9700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f03bc0a1150 0x7f03bc0a3610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009df0 tx=0x7f03c0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:43.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.725+0000 7f03c77fe700 1 -- 172.21.15.57:0/3067460435 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f03cc097cb0 con 0x7f03d0101310 2023-07-20T18:57:43.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.941+0000 7f03d830e700 1 -- 172.21.15.57:0/3067460435 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f03d002c970 con 0x7f03d0101310 2023-07-20T18:57:43.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.942+0000 7f03c77fe700 1 -- 172.21.15.57:0/3067460435 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7f03cc09b2d0 con 0x7f03d0101310 2023-07-20T18:57:43.991 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:57:43.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.944+0000 7f03d830e700 1 -- 172.21.15.57:0/3067460435 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f03bc0a1150 msgr2=0x7f03bc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:43.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.944+0000 7f03d830e700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f03bc0a1150 0x7f03bc0a3610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009df0 tx=0x7f03c0009260 comp rx=0 tx=0).stop 2023-07-20T18:57:43.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.944+0000 7f03d830e700 1 -- 172.21.15.57:0/3067460435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d0101310 msgr2=0x7f03d0100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:43.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.944+0000 7f03d830e700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d0101310 0x7f03d0100c70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f03cc000c00 tx=0x7f03cc0057c0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.944+0000 7f03d830e700 1 -- 172.21.15.57:0/3067460435 shutdown_connections 2023-07-20T18:57:43.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.944+0000 7f03d830e700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f03bc0a1150 0x7f03bc0a3610 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.944+0000 7f03d830e700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d0101310 0x7f03d0100c70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.944+0000 7f03d830e700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d0106570 0x7f03d00ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.944+0000 7f03d830e700 1 --2- 172.21.15.57:0/3067460435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d0103c40 0x7f03d00ff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:43.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.944+0000 7f03d830e700 1 -- 172.21.15.57:0/3067460435 >> 172.21.15.57:0/3067460435 conn(0x7f03d00fac80 msgr2=0x7f03d0101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:43.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.945+0000 7f03d830e700 1 -- 172.21.15.57:0/3067460435 shutdown_connections 2023-07-20T18:57:43.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:43.945+0000 7f03d830e700 1 -- 172.21.15.57:0/3067460435 wait complete. 2023-07-20T18:57:44.088 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:57:43.829+0000 7f74c1c42700 -1 osd.11 0 waiting for initial osdmap 2023-07-20T18:57:44.089 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:57:43.843+0000 7f74bd237700 -1 osd.11 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T18:57:44.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:44.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T18:57:44.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T18:57:44.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:44.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='osd.11 [v2:172.21.15.192:6824/2864190031,v1:172.21.15.192:6825/2864190031]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:57:44.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:57:44.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:44.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:44.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:43 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:44.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T18:57:44.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T18:57:44.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:44.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='osd.11 [v2:172.21.15.192:6824/2864190031,v1:172.21.15.192:6825/2864190031]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:57:44.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:57:44.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:44.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:44.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:43 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.141 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:44.142 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T18:57:44.142 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T18:57:44.142 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:44.142 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='osd.11 [v2:172.21.15.192:6824/2864190031,v1:172.21.15.192:6825/2864190031]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:57:44.142 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-20T18:57:44.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T18:57:44.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:44.144 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:44.144 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:43 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:44.421 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":76,"num_osds":12,"num_up_osds":11,"osd_up_since":1689879443,"num_in_osds":12,"osd_in_since":1689879447,"num_remapped_pgs":0} 2023-07-20T18:57:45.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:44 smithi057 ceph-mon[126340]: Detected new or changed devices on smithi192 2023-07-20T18:57:45.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:44 smithi057 ceph-mon[126340]: Adjusting osd_memory_target on smithi192 to 4292M 2023-07-20T18:57:45.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:44 smithi057 ceph-mon[126340]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:57:45.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:44 smithi057 ceph-mon[126340]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T18:57:45.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:44 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:45.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:44 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:45.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:44 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3067460435' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T18:57:45.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:44 smithi192 ceph-mon[129200]: Detected new or changed devices on smithi192 2023-07-20T18:57:45.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:44 smithi192 ceph-mon[129200]: Adjusting osd_memory_target on smithi192 to 4292M 2023-07-20T18:57:45.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:44 smithi192 ceph-mon[129200]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:57:45.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:44 smithi192 ceph-mon[129200]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T18:57:45.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:44 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:45.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:44 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:45.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:44 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3067460435' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T18:57:45.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:44 smithi088 ceph-mon[128823]: Detected new or changed devices on smithi192 2023-07-20T18:57:45.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:44 smithi088 ceph-mon[128823]: Adjusting osd_memory_target on smithi192 to 4292M 2023-07-20T18:57:45.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:44 smithi088 ceph-mon[128823]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-20T18:57:45.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:44 smithi088 ceph-mon[128823]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T18:57:45.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:44 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:45.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:44 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:45.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:44 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3067460435' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T18:57:45.422 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T18:57:46.169 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:57:46.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:45 smithi057 ceph-mon[126340]: purged_snaps scrub starts 2023-07-20T18:57:46.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:45 smithi057 ceph-mon[126340]: purged_snaps scrub ok 2023-07-20T18:57:46.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:45 smithi057 ceph-mon[126340]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:46.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:45 smithi057 ceph-mon[126340]: osd.11 [v2:172.21.15.192:6824/2864190031,v1:172.21.15.192:6825/2864190031] boot 2023-07-20T18:57:46.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:45 smithi057 ceph-mon[126340]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T18:57:46.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:45 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:46.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:45 smithi192 ceph-mon[129200]: purged_snaps scrub starts 2023-07-20T18:57:46.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:45 smithi192 ceph-mon[129200]: purged_snaps scrub ok 2023-07-20T18:57:46.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:45 smithi192 ceph-mon[129200]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:46.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:45 smithi192 ceph-mon[129200]: osd.11 [v2:172.21.15.192:6824/2864190031,v1:172.21.15.192:6825/2864190031] boot 2023-07-20T18:57:46.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:45 smithi192 ceph-mon[129200]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T18:57:46.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:45 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:46.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:45 smithi088 ceph-mon[128823]: purged_snaps scrub starts 2023-07-20T18:57:46.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:45 smithi088 ceph-mon[128823]: purged_snaps scrub ok 2023-07-20T18:57:46.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:45 smithi088 ceph-mon[128823]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T18:57:46.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:45 smithi088 ceph-mon[128823]: osd.11 [v2:172.21.15.192:6824/2864190031,v1:172.21.15.192:6825/2864190031] boot 2023-07-20T18:57:46.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:45 smithi088 ceph-mon[128823]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T18:57:46.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:45 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T18:57:46.984 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:46 smithi057 ceph-mon[126340]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T18:57:47.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:46 smithi192 ceph-mon[129200]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T18:57:47.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:46 smithi088 ceph-mon[128823]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T18:57:47.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.490+0000 7f87ab818700 1 -- 172.21.15.57:0/616930428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87a40747a0 msgr2=0x7f87a4072c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:47.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.490+0000 7f87ab818700 1 --2- 172.21.15.57:0/616930428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87a40747a0 0x7f87a4072c00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f87a00098f0 tx=0x7f87a0009c00 comp rx=0 tx=0).stop 2023-07-20T18:57:47.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.490+0000 7f87ab818700 1 -- 172.21.15.57:0/616930428 shutdown_connections 2023-07-20T18:57:47.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.490+0000 7f87ab818700 1 --2- 172.21.15.57:0/616930428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87a41086d0 0x7f87a410ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:47.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.490+0000 7f87ab818700 1 --2- 172.21.15.57:0/616930428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87a40747a0 0x7f87a4072c00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:47.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.490+0000 7f87ab818700 1 --2- 172.21.15.57:0/616930428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a40731d0 0x7f87a4073650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:47.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.490+0000 7f87ab818700 1 -- 172.21.15.57:0/616930428 >> 172.21.15.57:0/616930428 conn(0x7f87a40fbd60 msgr2=0x7f87a40fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:47.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.491+0000 7f87ab818700 1 -- 172.21.15.57:0/616930428 shutdown_connections 2023-07-20T18:57:47.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.491+0000 7f87ab818700 1 -- 172.21.15.57:0/616930428 wait complete. 2023-07-20T18:57:47.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.492+0000 7f87ab818700 1 Processor -- start 2023-07-20T18:57:47.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.492+0000 7f87ab818700 1 -- start start 2023-07-20T18:57:47.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87ab818700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a40731d0 0x7f87a4198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:47.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87ab818700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87a40747a0 0x7f87a4198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:47.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87ab818700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87a41086d0 0x7f87a419fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:47.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87ab818700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a41a03f0 con 0x7f87a40747a0 2023-07-20T18:57:47.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87ab818700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a41a0530 con 0x7f87a41086d0 2023-07-20T18:57:47.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87ab818700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a41a06c0 con 0x7f87a40731d0 2023-07-20T18:57:47.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87a8db3700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87a40747a0 0x7f87a4198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:47.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87a95b4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a40731d0 0x7f87a4198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:47.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87a8db3700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87a40747a0 0x7f87a4198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40642/0 (socket says 172.21.15.57:40642) 2023-07-20T18:57:47.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87a8db3700 1 -- 172.21.15.57:0/140132814 learned_addr learned my addr 172.21.15.57:0/140132814 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:57:47.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.493+0000 7f87a9db5700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87a41086d0 0x7f87a419fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:47.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f87a8db3700 1 -- 172.21.15.57:0/140132814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a40731d0 msgr2=0x7f87a4198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:47.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f87a8db3700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a40731d0 0x7f87a4198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:47.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f87a8db3700 1 -- 172.21.15.57:0/140132814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87a41086d0 msgr2=0x7f87a419fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:47.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f87a8db3700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87a41086d0 0x7f87a419fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:47.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f87a8db3700 1 -- 172.21.15.57:0/140132814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a0009520 con 0x7f87a40747a0 2023-07-20T18:57:47.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f87a95b4700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a40731d0 0x7f87a4198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:57:47.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f87a8db3700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87a40747a0 0x7f87a4198da0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f879400ea20 tx=0x7f879400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:47.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f879a7fc700 1 -- 172.21.15.57:0/140132814 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8794004220 con 0x7f87a40747a0 2023-07-20T18:57:47.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f87ab818700 1 -- 172.21.15.57:0/140132814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a41a09a0 con 0x7f87a40747a0 2023-07-20T18:57:47.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f879a7fc700 1 -- 172.21.15.57:0/140132814 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f8794004380 con 0x7f87a40747a0 2023-07-20T18:57:47.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.494+0000 7f879a7fc700 1 -- 172.21.15.57:0/140132814 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8794003be0 con 0x7f87a40747a0 2023-07-20T18:57:47.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.495+0000 7f87ab818700 1 -- 172.21.15.57:0/140132814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a41a0ea0 con 0x7f87a40747a0 2023-07-20T18:57:47.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.496+0000 7f87ab818700 1 -- 172.21.15.57:0/140132814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a4106d80 con 0x7f87a40747a0 2023-07-20T18:57:47.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.498+0000 7f879a7fc700 1 -- 172.21.15.57:0/140132814 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f879400c7c0 con 0x7f87a40747a0 2023-07-20T18:57:47.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.499+0000 7f879a7fc700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f87900a1150 0x7f87900a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:47.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.499+0000 7f879a7fc700 1 -- 172.21.15.57:0/140132814 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8794014070 con 0x7f87a40747a0 2023-07-20T18:57:47.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.499+0000 7f87a95b4700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f87900a1150 0x7f87900a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:47.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.500+0000 7f87a95b4700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f87900a1150 0x7f87900a3610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f87a0009fd0 tx=0x7f87a0019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:47.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.503+0000 7f879a7fc700 1 -- 172.21.15.57:0/140132814 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8794096130 con 0x7f87a40747a0 2023-07-20T18:57:47.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.711+0000 7f87ab818700 1 -- 172.21.15.57:0/140132814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f87a402c970 con 0x7f87a40747a0 2023-07-20T18:57:47.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.712+0000 7f879a7fc700 1 -- 172.21.15.57:0/140132814 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) v1 ==== 74+0+133 (secure 0 0 0) 0x7f8794099750 con 0x7f87a40747a0 2023-07-20T18:57:47.762 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:57:47.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 -- 172.21.15.57:0/140132814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f87900a1150 msgr2=0x7f87900a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:47.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f87900a1150 0x7f87900a3610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f87a0009fd0 tx=0x7f87a0019040 comp rx=0 tx=0).stop 2023-07-20T18:57:47.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 -- 172.21.15.57:0/140132814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87a40747a0 msgr2=0x7f87a4198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:47.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87a40747a0 0x7f87a4198da0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f879400ea20 tx=0x7f879400ed30 comp rx=0 tx=0).stop 2023-07-20T18:57:47.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 -- 172.21.15.57:0/140132814 shutdown_connections 2023-07-20T18:57:47.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f87900a1150 0x7f87900a3610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:47.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a40731d0 0x7f87a4198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:47.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87a41086d0 0x7f87a419fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:47.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 --2- 172.21.15.57:0/140132814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87a40747a0 0x7f87a4198da0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:47.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 -- 172.21.15.57:0/140132814 >> 172.21.15.57:0/140132814 conn(0x7f87a40fbd60 msgr2=0x7f87a40fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:47.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 -- 172.21.15.57:0/140132814 shutdown_connections 2023-07-20T18:57:47.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:47.714+0000 7f87ab818700 1 -- 172.21.15.57:0/140132814 wait complete. 2023-07-20T18:57:48.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:47 smithi057 ceph-mon[126340]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:48.231 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:47 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/140132814' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T18:57:48.251 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":78,"num_osds":12,"num_up_osds":12,"osd_up_since":1689879464,"num_in_osds":12,"osd_in_since":1689879447,"num_remapped_pgs":0} 2023-07-20T18:57:48.251 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph osd dump --format=json 2023-07-20T18:57:48.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:47 smithi192 ceph-mon[129200]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:48.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:47 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/140132814' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T18:57:48.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:47 smithi088 ceph-mon[128823]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:48.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:47 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/140132814' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T18:57:48.985 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:57:50.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:49 smithi057 ceph-mon[126340]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:50.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:49 smithi192 ceph-mon[129200]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:50.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:49 smithi088 ceph-mon[128823]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.406+0000 7f0429978700 1 -- 172.21.15.57:0/2770318011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0424100790 msgr2=0x7f0424100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:50.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.406+0000 7f0429978700 1 --2- 172.21.15.57:0/2770318011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0424100790 0x7f0424100c10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0418009990 tx=0x7f0418009ca0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.406+0000 7f0429978700 1 -- 172.21.15.57:0/2770318011 shutdown_connections 2023-07-20T18:57:50.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.406+0000 7f0429978700 1 --2- 172.21.15.57:0/2770318011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0424101150 0x7f042410d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.406+0000 7f0429978700 1 --2- 172.21.15.57:0/2770318011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0424106790 0x7f0424106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.406+0000 7f0429978700 1 --2- 172.21.15.57:0/2770318011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0424100790 0x7f0424100c10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.406+0000 7f0429978700 1 -- 172.21.15.57:0/2770318011 >> 172.21.15.57:0/2770318011 conn(0x7f04240fc260 msgr2=0x7f04240fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:50.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.406+0000 7f0429978700 1 -- 172.21.15.57:0/2770318011 shutdown_connections 2023-07-20T18:57:50.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.406+0000 7f0429978700 1 -- 172.21.15.57:0/2770318011 wait complete. 2023-07-20T18:57:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.407+0000 7f0429978700 1 Processor -- start 2023-07-20T18:57:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.408+0000 7f0429978700 1 -- start start 2023-07-20T18:57:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.408+0000 7f0429978700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0424100790 0x7f0424072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.408+0000 7f0429978700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0424101150 0x7f0424072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.408+0000 7f0429978700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0424106790 0x7f0424073530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.408+0000 7f0429978700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04241a4e20 con 0x7f0424100790 2023-07-20T18:57:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.408+0000 7f0429978700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04241a4f90 con 0x7f0424101150 2023-07-20T18:57:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.408+0000 7f0429978700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04241a5150 con 0x7f0424106790 2023-07-20T18:57:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.408+0000 7f0422ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0424100790 0x7f0424072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.408+0000 7f0422ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0424100790 0x7f0424072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40646/0 (socket says 172.21.15.57:40646) 2023-07-20T18:57:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.408+0000 7f0422ffd700 1 -- 172.21.15.57:0/522303736 learned_addr learned my addr 172.21.15.57:0/522303736 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:57:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f04227fc700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0424101150 0x7f0424072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f04237fe700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0424106790 0x7f0424073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f0422ffd700 1 -- 172.21.15.57:0/522303736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0424106790 msgr2=0x7f0424073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f0422ffd700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0424106790 0x7f0424073530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f0422ffd700 1 -- 172.21.15.57:0/522303736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0424101150 msgr2=0x7f0424072fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f0422ffd700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0424101150 0x7f0424072fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f0422ffd700 1 -- 172.21.15.57:0/522303736 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04180095f0 con 0x7f0424100790 2023-07-20T18:57:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f04227fc700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0424101150 0x7f0424072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:57:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f0422ffd700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0424100790 0x7f0424072a70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f041400e9a0 tx=0x7f041400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f0428976700 1 -- 172.21.15.57:0/522303736 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0414009790 con 0x7f0424100790 2023-07-20T18:57:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f0428976700 1 -- 172.21.15.57:0/522303736 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f0414010430 con 0x7f0424100790 2023-07-20T18:57:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f0429978700 1 -- 172.21.15.57:0/522303736 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04241a5430 con 0x7f0424100790 2023-07-20T18:57:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.409+0000 7f0428976700 1 -- 172.21.15.57:0/522303736 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0414018680 con 0x7f0424100790 2023-07-20T18:57:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.410+0000 7f0429978700 1 -- 172.21.15.57:0/522303736 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04241a5930 con 0x7f0424100790 2023-07-20T18:57:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.411+0000 7f0429978700 1 -- 172.21.15.57:0/522303736 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042404e470 con 0x7f0424100790 2023-07-20T18:57:50.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.413+0000 7f0428976700 1 -- 172.21.15.57:0/522303736 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0414010910 con 0x7f0424100790 2023-07-20T18:57:50.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.414+0000 7f0428976700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f040c0a1150 0x7f040c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:50.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.414+0000 7f0428976700 1 -- 172.21.15.57:0/522303736 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0414014070 con 0x7f0424100790 2023-07-20T18:57:50.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.414+0000 7f04227fc700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f040c0a1150 0x7f040c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:50.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.415+0000 7f04227fc700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f040c0a1150 0x7f040c0a3610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0424074260 tx=0x7f0418005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:50.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.418+0000 7f0428976700 1 -- 172.21.15.57:0/522303736 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0414051780 con 0x7f0424100790 2023-07-20T18:57:50.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.627+0000 7f0429978700 1 -- 172.21.15.57:0/522303736 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04241a5c50 con 0x7f0424100790 2023-07-20T18:57:50.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.629+0000 7f0428976700 1 -- 172.21.15.57:0/522303736 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18373 (secure 0 0 0) 0x7f0414096210 con 0x7f0424100790 2023-07-20T18:57:50.678 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:57:50.678 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":78,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","created":"2023-07-20T18:51:01.245641+0000","modified":"2023-07-20T18:57:45.836366+0000","last_up_change":"2023-07-20T18:57:44.823544+0000","last_in_change":"2023-07-20T18:57:27.428782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T18:54:23.215777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"627a67ea-1366-4b3f-babd-067f75bad1e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6803","nonce":3023441607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6805","nonce":3023441607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6809","nonce":3023441607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6807","nonce":3023441607}]},"public_addr":"172.21.15.57:6803/3023441607","cluster_addr":"172.21.15.57:6805/3023441607","heartbeat_back_addr":"172.21.15.57:6809/3023441607","heartbeat_front_addr":"172.21.15.57:6807/3023441607","state":["exists","up"]},{"osd":1,"uuid":"adcdb236-334e-4478-98a8-7d0628492082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6801","nonce":1809528555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6803","nonce":1809528555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6807","nonce":1809528555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6805","nonce":1809528555}]},"public_addr":"172.21.15.88:6801/1809528555","cluster_addr":"172.21.15.88:6803/1809528555","heartbeat_back_addr":"172.21.15.88:6807/1809528555","heartbeat_front_addr":"172.21.15.88:6805/1809528555","state":["exists","up"]},{"osd":2,"uuid":"3cd56229-2067-433a-b9f9-f47b1b878b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6801","nonce":3891687382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6803","nonce":3891687382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6807","nonce":3891687382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6805","nonce":3891687382}]},"public_addr":"172.21.15.192:6801/3891687382","cluster_addr":"172.21.15.192:6803/3891687382","heartbeat_back_addr":"172.21.15.192:6807/3891687382","heartbeat_front_addr":"172.21.15.192:6805/3891687382","state":["exists","up"]},{"osd":3,"uuid":"aae250ab-e242-4ef8-9c3a-a4fa36102c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6811","nonce":2920717314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6813","nonce":2920717314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6817","nonce":2920717314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6815","nonce":2920717314}]},"public_addr":"172.21.15.57:6811/2920717314","cluster_addr":"172.21.15.57:6813/2920717314","heartbeat_back_addr":"172.21.15.57:6817/2920717314","heartbeat_front_addr":"172.21.15.57:6815/2920717314","state":["exists","up"]},{"osd":4,"uuid":"8c8d1dfa-1cff-4f0a-ae72-22bd522ed696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6809","nonce":3911525610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6811","nonce":3911525610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6815","nonce":3911525610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6813","nonce":3911525610}]},"public_addr":"172.21.15.88:6809/3911525610","cluster_addr":"172.21.15.88:6811/3911525610","heartbeat_back_addr":"172.21.15.88:6815/3911525610","heartbeat_front_addr":"172.21.15.88:6813/3911525610","state":["exists","up"]},{"osd":5,"uuid":"5a2a56ab-2af9-4176-ba4d-b2138f0e7c75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6809","nonce":905436397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6811","nonce":905436397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6815","nonce":905436397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6813","nonce":905436397}]},"public_addr":"172.21.15.192:6809/905436397","cluster_addr":"172.21.15.192:6811/905436397","heartbeat_back_addr":"172.21.15.192:6815/905436397","heartbeat_front_addr":"172.21.15.192:6813/905436397","state":["exists","up"]},{"osd":6,"uuid":"03eafdda-278c-443c-b376-677eeb50262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6819","nonce":333894423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6821","nonce":333894423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6825","nonce":333894423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6823","nonce":333894423}]},"public_addr":"172.21.15.57:6819/333894423","cluster_addr":"172.21.15.57:6821/333894423","heartbeat_back_addr":"172.21.15.57:6825/333894423","heartbeat_front_addr":"172.21.15.57:6823/333894423","state":["exists","up"]},{"osd":7,"uuid":"3f01254e-be73-41c8-b979-3b6a7534e2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6817","nonce":3031485551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6819","nonce":3031485551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6823","nonce":3031485551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6821","nonce":3031485551}]},"public_addr":"172.21.15.88:6817/3031485551","cluster_addr":"172.21.15.88:6819/3031485551","heartbeat_back_addr":"172.21.15.88:6823/3031485551","heartbeat_front_addr":"172.21.15.88:6821/3031485551","state":["exists","up"]},{"osd":8,"uuid":"624a150e-f47b-4a76-8785-c818dd65342e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6817","nonce":918056390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6819","nonce":918056390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6823","nonce":918056390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6821","nonce":918056390}]},"public_addr":"172.21.15.192:6817/918056390","cluster_addr":"172.21.15.192:6819/918056390","heartbeat_back_addr":"172.21.15.192:6823/918056390","heartbeat_front_addr":"172.21.15.192:6821/918056390","state":["exists","up"]},{"osd":9,"uuid":"6d9cb69d-ed7b-4009-9ec5-027becc577d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6827","nonce":1773340763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6829","nonce":1773340763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6833","nonce":1773340763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6831","nonce":1773340763}]},"public_addr":"172.21.15.57:6827/1773340763","cluster_addr":"172.21.15.57:6829/1773340763","heartbeat_back_addr":"172.21.15.57:6833/1773340763","heartbeat_front_addr":"172.21.15.57:6831/1773340763","state":["exists","up"]},{"osd":10,"uuid":"d5799219-0979-40b1-a596-acd37f37a72d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6825","nonce":2781975579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6827","nonce":2781975579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6831","nonce":2781975579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6829","nonce":2781975579}]},"public_addr":"172.21.15.88:6825/2781975579","cluster_addr":"172.21.15.88:6827/2781975579","heartbeat_back_addr":"172.21.15.88:6831/2781975579","heartbeat_front_addr":"172.21.15.88:6829/2781975579","state":["exists","up"]},{"osd":11,"uuid":"e77cac6c-107e-41f9-8efd-e1140c4e3bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6825","nonce":2864190031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6827","nonce":2864190031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6831","nonce":2864190031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6829","nonce":2864190031}]},"public_addr":"172.21.15.192:6825/2864190031","cluster_addr":"172.21.15.192:6827/2864190031","heartbeat_back_addr":"172.21.15.192:6831/2864190031","heartbeat_front_addr":"172.21.15.192:6829/2864190031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:53:38.180971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:00.253039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:20.776122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:48.631094+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:09.288231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:29.595802+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:55.089126+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:15.637849+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:35.985101+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:01.544767+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:22.351743+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:43.255157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6801/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/3804015220":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6800/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1582727629":"2023-07-21T18:51:34.614904+0000","172.21.15.57:6800/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/2044626900":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/748511752":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1151100650":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6801/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/3446341201":"2023-07-21T18:52:02.610729+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T18:57:50.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 -- 172.21.15.57:0/522303736 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f040c0a1150 msgr2=0x7f040c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:50.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f040c0a1150 0x7f040c0a3610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0424074260 tx=0x7f0418005fd0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 -- 172.21.15.57:0/522303736 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0424100790 msgr2=0x7f0424072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:50.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0424100790 0x7f0424072a70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f041400e9a0 tx=0x7f041400ecb0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 -- 172.21.15.57:0/522303736 shutdown_connections 2023-07-20T18:57:50.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f040c0a1150 0x7f040c0a3610 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0424106790 0x7f0424073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0424100790 0x7f0424072a70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 --2- 172.21.15.57:0/522303736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0424101150 0x7f0424072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:50.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 -- 172.21.15.57:0/522303736 >> 172.21.15.57:0/522303736 conn(0x7f04240fc260 msgr2=0x7f04240fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:50.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 -- 172.21.15.57:0/522303736 shutdown_connections 2023-07-20T18:57:50.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:50.631+0000 7f0429978700 1 -- 172.21.15.57:0/522303736 wait complete. 2023-07-20T18:57:50.984 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:50 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/522303736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:57:51.165 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T18:54:23.215777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T18:57:51.166 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-20T18:57:51.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:50 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/522303736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:57:51.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:50 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/522303736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:57:51.898 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:57:52.186 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:51 smithi057 ceph-mon[126340]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:52.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:51 smithi192 ceph-mon[129200]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:52.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:51 smithi088 ceph-mon[128823]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.292+0000 7f37bbc0f700 1 -- 172.21.15.57:0/12032091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37b4106790 msgr2=0x7f37b4106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.292+0000 7f37bbc0f700 1 --2- 172.21.15.57:0/12032091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37b4106790 0x7f37b4106b70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f37b00098f0 tx=0x7f37b0009c00 comp rx=0 tx=0).stop 2023-07-20T18:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.292+0000 7f37bbc0f700 1 -- 172.21.15.57:0/12032091 shutdown_connections 2023-07-20T18:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.292+0000 7f37bbc0f700 1 --2- 172.21.15.57:0/12032091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37b4106790 0x7f37b4106b70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.292+0000 7f37bbc0f700 1 --2- 172.21.15.57:0/12032091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b4101150 0x7f37b410d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.292+0000 7f37bbc0f700 1 --2- 172.21.15.57:0/12032091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b4100790 0x7f37b4100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.292+0000 7f37bbc0f700 1 -- 172.21.15.57:0/12032091 >> 172.21.15.57:0/12032091 conn(0x7f37b40fc260 msgr2=0x7f37b40fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:53.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.292+0000 7f37bbc0f700 1 -- 172.21.15.57:0/12032091 shutdown_connections 2023-07-20T18:57:53.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.292+0000 7f37bbc0f700 1 -- 172.21.15.57:0/12032091 wait complete. 2023-07-20T18:57:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.293+0000 7f37bbc0f700 1 Processor -- start 2023-07-20T18:57:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.293+0000 7f37bbc0f700 1 -- start start 2023-07-20T18:57:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37bbc0f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b4100790 0x7f37b419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37bbc0f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b4101150 0x7f37b419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37bbc0f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37b4106790 0x7f37b41989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37bbc0f700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b4199040 con 0x7f37b4101150 2023-07-20T18:57:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37bbc0f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b41991b0 con 0x7f37b4100790 2023-07-20T18:57:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37bbc0f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b4199370 con 0x7f37b4106790 2023-07-20T18:57:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37b91aa700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b4101150 0x7f37b419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37b91aa700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b4101150 0x7f37b419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40662/0 (socket says 172.21.15.57:40662) 2023-07-20T18:57:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37b91aa700 1 -- 172.21.15.57:0/1042655534 learned_addr learned my addr 172.21.15.57:0/1042655534 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:57:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37b99ab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b4100790 0x7f37b419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.294+0000 7f37ba1ac700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37b4106790 0x7f37b41989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37b99ab700 1 -- 172.21.15.57:0/1042655534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37b4106790 msgr2=0x7f37b41989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37b99ab700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37b4106790 0x7f37b41989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37b99ab700 1 -- 172.21.15.57:0/1042655534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b4101150 msgr2=0x7f37b419ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37b99ab700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b4101150 0x7f37b419ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37b99ab700 1 -- 172.21.15.57:0/1042655534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac009d00 con 0x7f37b4100790 2023-07-20T18:57:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37b91aa700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b4101150 0x7f37b419ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:57:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37b99ab700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b4100790 0x7f37b419e950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009230 tx=0x7f37b0005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:53.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37aaffd700 1 -- 172.21.15.57:0/1042655534 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b001c070 con 0x7f37b4100790 2023-07-20T18:57:53.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37aaffd700 1 -- 172.21.15.57:0/1042655534 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f37b0005690 con 0x7f37b4100790 2023-07-20T18:57:53.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37bbc0f700 1 -- 172.21.15.57:0/1042655534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b0009520 con 0x7f37b4100790 2023-07-20T18:57:53.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37aaffd700 1 -- 172.21.15.57:0/1042655534 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b00039e0 con 0x7f37b4100790 2023-07-20T18:57:53.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.295+0000 7f37bbc0f700 1 -- 172.21.15.57:0/1042655534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b41a5770 con 0x7f37b4100790 2023-07-20T18:57:53.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.296+0000 7f37bbc0f700 1 -- 172.21.15.57:0/1042655534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b404e470 con 0x7f37b4100790 2023-07-20T18:57:53.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.298+0000 7f37aaffd700 1 -- 172.21.15.57:0/1042655534 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f37b0020030 con 0x7f37b4100790 2023-07-20T18:57:53.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.300+0000 7f37aaffd700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f37a00a11d0 0x7f37a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:53.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.300+0000 7f37aaffd700 1 -- 172.21.15.57:0/1042655534 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f37b00cd2c0 con 0x7f37b4100790 2023-07-20T18:57:53.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.300+0000 7f37b91aa700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f37a00a11d0 0x7f37a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:53.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.300+0000 7f37b91aa700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f37a00a11d0 0x7f37a00a3690 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f37b410da90 tx=0x7f37a4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:53.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.304+0000 7f37aaffd700 1 -- 172.21.15.57:0/1042655534 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f37b0096ea0 con 0x7f37b4100790 2023-07-20T18:57:53.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.510+0000 7f37bbc0f700 1 -- 172.21.15.57:0/1042655534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37b41a59f0 con 0x7f37b4100790 2023-07-20T18:57:53.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.511+0000 7f37aaffd700 1 -- 172.21.15.57:0/1042655534 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37b009a4c0 con 0x7f37b4100790 2023-07-20T18:57:53.560 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-07-20T18:57:53.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.513+0000 7f37bbc0f700 1 -- 172.21.15.57:0/1042655534 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f37a00a11d0 msgr2=0x7f37a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:53.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.513+0000 7f37bbc0f700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f37a00a11d0 0x7f37a00a3690 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f37b410da90 tx=0x7f37a4009260 comp rx=0 tx=0).stop 2023-07-20T18:57:53.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.513+0000 7f37bbc0f700 1 -- 172.21.15.57:0/1042655534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b4100790 msgr2=0x7f37b419e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:53.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.513+0000 7f37bbc0f700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b4100790 0x7f37b419e950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009230 tx=0x7f37b0005040 comp rx=0 tx=0).stop 2023-07-20T18:57:53.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.514+0000 7f37bbc0f700 1 -- 172.21.15.57:0/1042655534 shutdown_connections 2023-07-20T18:57:53.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.514+0000 7f37bbc0f700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f37a00a11d0 0x7f37a00a3690 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:53.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.514+0000 7f37bbc0f700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37b4106790 0x7f37b41989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:53.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.514+0000 7f37bbc0f700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b4100790 0x7f37b419e950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:53.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.514+0000 7f37bbc0f700 1 --2- 172.21.15.57:0/1042655534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b4101150 0x7f37b419ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:53.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.514+0000 7f37bbc0f700 1 -- 172.21.15.57:0/1042655534 >> 172.21.15.57:0/1042655534 conn(0x7f37b40fc260 msgr2=0x7f37b410bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:53.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.514+0000 7f37bbc0f700 1 -- 172.21.15.57:0/1042655534 shutdown_connections 2023-07-20T18:57:53.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:53.514+0000 7f37bbc0f700 1 -- 172.21.15.57:0/1042655534 wait complete. 2023-07-20T18:57:53.982 INFO:tasks.cephadm:Adding mds.a on smithi057 2023-07-20T18:57:53.983 INFO:tasks.cephadm:Adding mds.d on smithi057 2023-07-20T18:57:53.983 INFO:tasks.cephadm:Adding mds.g on smithi057 2023-07-20T18:57:53.983 INFO:tasks.cephadm:Adding mds.j on smithi057 2023-07-20T18:57:53.983 INFO:tasks.cephadm:Adding mds.b on smithi088 2023-07-20T18:57:53.984 INFO:tasks.cephadm:Adding mds.e on smithi088 2023-07-20T18:57:53.984 INFO:tasks.cephadm:Adding mds.h on smithi088 2023-07-20T18:57:53.984 INFO:tasks.cephadm:Adding mds.k on smithi088 2023-07-20T18:57:53.984 INFO:tasks.cephadm:Adding mds.c on smithi192 2023-07-20T18:57:53.984 INFO:tasks.cephadm:Adding mds.f on smithi192 2023-07-20T18:57:53.985 INFO:tasks.cephadm:Adding mds.i on smithi192 2023-07-20T18:57:53.985 INFO:tasks.cephadm:Adding mds.l on smithi192 2023-07-20T18:57:53.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph orch apply mds all '12;smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l' 2023-07-20T18:57:54.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:53 smithi057 ceph-mon[126340]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:54.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:53 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1042655534' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:57:54.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:53 smithi192 ceph-mon[129200]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:54.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:53 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1042655534' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:57:54.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:53 smithi088 ceph-mon[128823]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:54.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:53 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1042655534' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:57:54.554 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T18:57:55.778 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.729+0000 7fd1377ca700 1 -- 172.21.15.192:0/3265556634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1300fee80 msgr2=0x7fd1300ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:55.778 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.729+0000 7fd1377ca700 1 --2- 172.21.15.192:0/3265556634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1300fee80 0x7fd1300ff2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd124009990 tx=0x7fd124009ca0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.779 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.729+0000 7fd1377ca700 1 -- 172.21.15.192:0/3265556634 shutdown_connections 2023-07-20T18:57:55.779 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.729+0000 7fd1377ca700 1 --2- 172.21.15.192:0/3265556634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1300fee80 0x7fd1300ff2a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.779 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.729+0000 7fd1377ca700 1 --2- 172.21.15.192:0/3265556634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1301009a0 0x7fd1301072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.779 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.729+0000 7fd1377ca700 1 --2- 172.21.15.192:0/3265556634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1300fffe0 0x7fd130100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.779 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.729+0000 7fd1377ca700 1 -- 172.21.15.192:0/3265556634 >> 172.21.15.192:0/3265556634 conn(0x7fd1300fa420 msgr2=0x7fd1300fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:55.779 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.729+0000 7fd1377ca700 1 -- 172.21.15.192:0/3265556634 shutdown_connections 2023-07-20T18:57:55.779 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.729+0000 7fd1377ca700 1 -- 172.21.15.192:0/3265556634 wait complete. 2023-07-20T18:57:55.780 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd1377ca700 1 Processor -- start 2023-07-20T18:57:55.780 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd1377ca700 1 -- start start 2023-07-20T18:57:55.781 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd1377ca700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1300fee80 0x7fd130195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:55.781 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd1377ca700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1300fffe0 0x7fd130195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:55.781 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd1377ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1301009a0 0x7fd13019ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:55.782 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd1377ca700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13019cfe0 con 0x7fd1300fffe0 2023-07-20T18:57:55.782 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd1377ca700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13019d120 con 0x7fd1300fee80 2023-07-20T18:57:55.782 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd1377ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13019d2b0 con 0x7fd1301009a0 2023-07-20T18:57:55.782 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd135d67700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1301009a0 0x7fd13019ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:55.782 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd135d67700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1301009a0 0x7fd13019ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:48608/0 (socket says 172.21.15.192:48608) 2023-07-20T18:57:55.782 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd135d67700 1 -- 172.21.15.192:0/1052295005 learned_addr learned my addr 172.21.15.192:0/1052295005 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-07-20T18:57:55.782 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd135566700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1300fee80 0x7fd130195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:55.783 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.730+0000 7fd134d65700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1300fffe0 0x7fd130195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:55.783 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd135d67700 1 -- 172.21.15.192:0/1052295005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1300fee80 msgr2=0x7fd130195450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:55.783 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd135d67700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1300fee80 0x7fd130195450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.783 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd135d67700 1 -- 172.21.15.192:0/1052295005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1300fffe0 msgr2=0x7fd130195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:55.783 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd135d67700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1300fffe0 0x7fd130195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.783 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd135d67700 1 -- 172.21.15.192:0/1052295005 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1240095f0 con 0x7fd1301009a0 2023-07-20T18:57:55.783 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd135d67700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1301009a0 0x7fd13019ca10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd12800dbc0 tx=0x7fd12800ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:55.783 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd1227fc700 1 -- 172.21.15.192:0/1052295005 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12800e710 con 0x7fd1301009a0 2023-07-20T18:57:55.784 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd1227fc700 1 -- 172.21.15.192:0/1052295005 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fd12800bac0 con 0x7fd1301009a0 2023-07-20T18:57:55.784 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd1377ca700 1 -- 172.21.15.192:0/1052295005 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1300fbc80 con 0x7fd1301009a0 2023-07-20T18:57:55.784 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd1227fc700 1 -- 172.21.15.192:0/1052295005 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1280163f0 con 0x7fd1301009a0 2023-07-20T18:57:55.784 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.731+0000 7fd1377ca700 1 -- 172.21.15.192:0/1052295005 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1300fc1d0 con 0x7fd1301009a0 2023-07-20T18:57:55.784 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.732+0000 7fd1377ca700 1 -- 172.21.15.192:0/1052295005 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13004e470 con 0x7fd1301009a0 2023-07-20T18:57:55.784 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.733+0000 7fd1227fc700 1 -- 172.21.15.192:0/1052295005 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd128009580 con 0x7fd1301009a0 2023-07-20T18:57:55.785 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.733+0000 7fd1227fc700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd11c0a11d0 0x7fd11c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:55.785 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.734+0000 7fd1227fc700 1 -- 172.21.15.192:0/1052295005 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd128006e40 con 0x7fd1301009a0 2023-07-20T18:57:55.785 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.734+0000 7fd135566700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd11c0a11d0 0x7fd11c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:55.785 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.734+0000 7fd135566700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd11c0a11d0 0x7fd11c0a3690 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd12400b5c0 tx=0x7fd124003680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:55.786 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.738+0000 7fd1227fc700 1 -- 172.21.15.192:0/1052295005 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd12804f780 con 0x7fd1301009a0 2023-07-20T18:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.933+0000 7fd1377ca700 1 -- 172.21.15.192:0/1052295005 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1300fc6b0 con 0x7fd11c0a11d0 2023-07-20T18:57:55.989 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.941+0000 7fd1227fc700 1 -- 172.21.15.192:0/1052295005 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fd1300fc6b0 con 0x7fd11c0a11d0 2023-07-20T18:57:55.989 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mds.all update... 2023-07-20T18:57:55.991 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 -- 172.21.15.192:0/1052295005 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd11c0a11d0 msgr2=0x7fd11c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:55.992 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd11c0a11d0 0x7fd11c0a3690 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd12400b5c0 tx=0x7fd124003680 comp rx=0 tx=0).stop 2023-07-20T18:57:55.992 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 -- 172.21.15.192:0/1052295005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1301009a0 msgr2=0x7fd13019ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:55.992 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1301009a0 0x7fd13019ca10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd12800dbc0 tx=0x7fd12800ded0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.992 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 -- 172.21.15.192:0/1052295005 shutdown_connections 2023-07-20T18:57:55.992 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd11c0a11d0 0x7fd11c0a3690 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.992 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1301009a0 0x7fd13019ca10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.993 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1300fee80 0x7fd130195450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.993 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 --2- 172.21.15.192:0/1052295005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1300fffe0 0x7fd130195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:55.993 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 -- 172.21.15.192:0/1052295005 >> 172.21.15.192:0/1052295005 conn(0x7fd1300fa420 msgr2=0x7fd130103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:55.993 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 -- 172.21.15.192:0/1052295005 shutdown_connections 2023-07-20T18:57:55.993 INFO:teuthology.orchestra.run.smithi192.stderr:2023-07-20T18:57:55.943+0000 7fd1377ca700 1 -- 172.21.15.192:0/1052295005 wait complete. 2023-07-20T18:57:56.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:55 smithi057 ceph-mon[126340]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:56.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:55 smithi192 ceph-mon[129200]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:56.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:55 smithi088 ceph-mon[128823]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:56.412 DEBUG:teuthology.orchestra.run.smithi057:mds.a> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.a.service 2023-07-20T18:57:56.417 DEBUG:teuthology.orchestra.run.smithi057:mds.d> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.d.service 2023-07-20T18:57:56.466 DEBUG:teuthology.orchestra.run.smithi057:mds.g> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.g.service 2023-07-20T18:57:56.500 INFO:journalctl@ceph.mds.a.smithi057.stdout:-- Logs begin at Thu 2023-07-20 18:30:45 UTC. -- 2023-07-20T18:57:56.503 DEBUG:teuthology.orchestra.run.smithi057:mds.j> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.j.service 2023-07-20T18:57:56.540 INFO:journalctl@ceph.mds.d.smithi057.stdout:-- Logs begin at Thu 2023-07-20 18:30:45 UTC. -- 2023-07-20T18:57:56.543 DEBUG:teuthology.orchestra.run.smithi088:mds.b> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.b.service 2023-07-20T18:57:56.546 DEBUG:teuthology.orchestra.run.smithi088:mds.e> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.e.service 2023-07-20T18:57:56.575 INFO:journalctl@ceph.mds.g.smithi057.stdout:-- Logs begin at Thu 2023-07-20 18:30:45 UTC. -- 2023-07-20T18:57:56.583 INFO:journalctl@ceph.mds.b.smithi088.stdout:-- Logs begin at Thu 2023-07-20 18:25:07 UTC. -- 2023-07-20T18:57:56.584 DEBUG:teuthology.orchestra.run.smithi088:mds.h> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.h.service 2023-07-20T18:57:56.612 INFO:journalctl@ceph.mds.j.smithi057.stdout:-- Logs begin at Thu 2023-07-20 18:30:45 UTC. -- 2023-07-20T18:57:56.613 INFO:journalctl@ceph.mds.e.smithi088.stdout:-- Logs begin at Thu 2023-07-20 18:25:07 UTC. -- 2023-07-20T18:57:56.614 DEBUG:teuthology.orchestra.run.smithi088:mds.k> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.k.service 2023-07-20T18:57:56.641 INFO:journalctl@ceph.mds.h.smithi088.stdout:-- Logs begin at Thu 2023-07-20 18:25:07 UTC. -- 2023-07-20T18:57:56.642 DEBUG:teuthology.orchestra.run.smithi192:mds.c> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.c.service 2023-07-20T18:57:56.645 DEBUG:teuthology.orchestra.run.smithi192:mds.f> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.f.service 2023-07-20T18:57:56.668 INFO:journalctl@ceph.mds.k.smithi088.stdout:-- Logs begin at Thu 2023-07-20 18:25:07 UTC. -- 2023-07-20T18:57:56.674 INFO:journalctl@ceph.mds.c.smithi192.stdout:-- Logs begin at Thu 2023-07-20 18:24:33 UTC. -- 2023-07-20T18:57:56.675 DEBUG:teuthology.orchestra.run.smithi192:mds.i> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.i.service 2023-07-20T18:57:56.702 INFO:journalctl@ceph.mds.f.smithi192.stdout:-- Logs begin at Thu 2023-07-20 18:24:33 UTC. -- 2023-07-20T18:57:56.703 DEBUG:teuthology.orchestra.run.smithi192:mds.l> sudo journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.l.service 2023-07-20T18:57:56.729 INFO:journalctl@ceph.mds.i.smithi192.stdout:-- Logs begin at Thu 2023-07-20 18:24:33 UTC. -- 2023-07-20T18:57:56.730 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-20T18:57:56.731 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T18:57:56.756 INFO:journalctl@ceph.mds.l.smithi192.stdout:-- Logs begin at Thu 2023-07-20 18:24:33 UTC. -- 2023-07-20T18:57:56.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.900+0000 7f90e73ff700 1 -- 172.21.15.57:0/2864274972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90e0101960 msgr2=0x7f90e0101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:56.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.900+0000 7f90e73ff700 1 --2- 172.21.15.57:0/2864274972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90e0101960 0x7f90e0101dc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009920 tx=0x7f90d0009c30 comp rx=0 tx=0).stop 2023-07-20T18:57:56.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.900+0000 7f90e73ff700 1 -- 172.21.15.57:0/2864274972 shutdown_connections 2023-07-20T18:57:56.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.900+0000 7f90e73ff700 1 --2- 172.21.15.57:0/2864274972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e0100770 0x7f90e0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:56.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.900+0000 7f90e73ff700 1 --2- 172.21.15.57:0/2864274972 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90e0102300 0x7f90e0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:56.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.900+0000 7f90e73ff700 1 --2- 172.21.15.57:0/2864274972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90e0101960 0x7f90e0101dc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:56.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.900+0000 7f90e73ff700 1 -- 172.21.15.57:0/2864274972 >> 172.21.15.57:0/2864274972 conn(0x7f90e00fbca0 msgr2=0x7f90e00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:56.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.901+0000 7f90e73ff700 1 -- 172.21.15.57:0/2864274972 shutdown_connections 2023-07-20T18:57:56.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.901+0000 7f90e73ff700 1 -- 172.21.15.57:0/2864274972 wait complete. 2023-07-20T18:57:56.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.901+0000 7f90e73ff700 1 Processor -- start 2023-07-20T18:57:56.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e73ff700 1 -- start start 2023-07-20T18:57:56.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e73ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90e0100770 0x7f90e0074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:56.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e73ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e0101960 0x7f90e0072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:56.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e73ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90e0102300 0x7f90e00730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:56.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e73ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e0073840 con 0x7f90e0102300 2023-07-20T18:57:56.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e73ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e00739b0 con 0x7f90e0100770 2023-07-20T18:57:56.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e73ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e01a0610 con 0x7f90e0101960 2023-07-20T18:57:56.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e499a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e0101960 0x7f90e0072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:56.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e599c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90e0102300 0x7f90e00730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:56.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e499a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e0101960 0x7f90e0072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:57318/0 (socket says 172.21.15.57:57318) 2023-07-20T18:57:56.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e499a700 1 -- 172.21.15.57:0/182168369 learned_addr learned my addr 172.21.15.57:0/182168369 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:57:56.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.902+0000 7f90e519b700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90e0100770 0x7f90e0074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:56.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90e499a700 1 -- 172.21.15.57:0/182168369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90e0100770 msgr2=0x7f90e0074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:56.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90e499a700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90e0100770 0x7f90e0074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:56.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90e499a700 1 -- 172.21.15.57:0/182168369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90e0102300 msgr2=0x7f90e00730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:56.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90e499a700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90e0102300 0x7f90e00730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:56.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90e499a700 1 -- 172.21.15.57:0/182168369 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc009c40 con 0x7f90e0101960 2023-07-20T18:57:56.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90e599c700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90e0102300 0x7f90e00730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:57:56.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90e499a700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e0101960 0x7f90e0072b60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009230 tx=0x7f90d0005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:56.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90d67fc700 1 -- 172.21.15.57:0/182168369 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d001c070 con 0x7f90e0101960 2023-07-20T18:57:56.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90e73ff700 1 -- 172.21.15.57:0/182168369 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d0009520 con 0x7f90e0101960 2023-07-20T18:57:56.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90d67fc700 1 -- 172.21.15.57:0/182168369 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f90d000f950 con 0x7f90e0101960 2023-07-20T18:57:56.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90d67fc700 1 -- 172.21.15.57:0/182168369 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d0003a60 con 0x7f90e0101960 2023-07-20T18:57:56.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.903+0000 7f90e73ff700 1 -- 172.21.15.57:0/182168369 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e01a0ad0 con 0x7f90e0101960 2023-07-20T18:57:56.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.904+0000 7f90e73ff700 1 -- 172.21.15.57:0/182168369 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e0107d30 con 0x7f90e0101960 2023-07-20T18:57:56.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.906+0000 7f90d67fc700 1 -- 172.21.15.57:0/182168369 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f90d0020030 con 0x7f90e0101960 2023-07-20T18:57:56.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.907+0000 7f90d67fc700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f90cc0a11a0 0x7f90cc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:56.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.907+0000 7f90d67fc700 1 -- 172.21.15.57:0/182168369 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f90d00cd480 con 0x7f90e0101960 2023-07-20T18:57:56.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.908+0000 7f90e519b700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f90cc0a11a0 0x7f90cc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:56.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.908+0000 7f90e519b700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f90cc0a11a0 0x7f90cc0a3660 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f90e01017c0 tx=0x7f90dc00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:56.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:56.911+0000 7f90d67fc700 1 -- 172.21.15.57:0/182168369 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f90d0097060 con 0x7f90e0101960 2023-07-20T18:57:57.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.122+0000 7f90e73ff700 1 -- 172.21.15.57:0/182168369 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90e002c970 con 0x7f90e0101960 2023-07-20T18:57:57.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.124+0000 7f90d67fc700 1 -- 172.21.15.57:0/182168369 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18373 (secure 0 0 0) 0x7f90d009a680 con 0x7f90e0101960 2023-07-20T18:57:57.173 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:57:57.174 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":78,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","created":"2023-07-20T18:51:01.245641+0000","modified":"2023-07-20T18:57:45.836366+0000","last_up_change":"2023-07-20T18:57:44.823544+0000","last_in_change":"2023-07-20T18:57:27.428782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T18:54:23.215777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"627a67ea-1366-4b3f-babd-067f75bad1e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6803","nonce":3023441607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6805","nonce":3023441607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6809","nonce":3023441607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6807","nonce":3023441607}]},"public_addr":"172.21.15.57:6803/3023441607","cluster_addr":"172.21.15.57:6805/3023441607","heartbeat_back_addr":"172.21.15.57:6809/3023441607","heartbeat_front_addr":"172.21.15.57:6807/3023441607","state":["exists","up"]},{"osd":1,"uuid":"adcdb236-334e-4478-98a8-7d0628492082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6801","nonce":1809528555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6803","nonce":1809528555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6807","nonce":1809528555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6805","nonce":1809528555}]},"public_addr":"172.21.15.88:6801/1809528555","cluster_addr":"172.21.15.88:6803/1809528555","heartbeat_back_addr":"172.21.15.88:6807/1809528555","heartbeat_front_addr":"172.21.15.88:6805/1809528555","state":["exists","up"]},{"osd":2,"uuid":"3cd56229-2067-433a-b9f9-f47b1b878b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6801","nonce":3891687382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6803","nonce":3891687382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6807","nonce":3891687382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6805","nonce":3891687382}]},"public_addr":"172.21.15.192:6801/3891687382","cluster_addr":"172.21.15.192:6803/3891687382","heartbeat_back_addr":"172.21.15.192:6807/3891687382","heartbeat_front_addr":"172.21.15.192:6805/3891687382","state":["exists","up"]},{"osd":3,"uuid":"aae250ab-e242-4ef8-9c3a-a4fa36102c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6811","nonce":2920717314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6813","nonce":2920717314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6817","nonce":2920717314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6815","nonce":2920717314}]},"public_addr":"172.21.15.57:6811/2920717314","cluster_addr":"172.21.15.57:6813/2920717314","heartbeat_back_addr":"172.21.15.57:6817/2920717314","heartbeat_front_addr":"172.21.15.57:6815/2920717314","state":["exists","up"]},{"osd":4,"uuid":"8c8d1dfa-1cff-4f0a-ae72-22bd522ed696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6809","nonce":3911525610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6811","nonce":3911525610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6815","nonce":3911525610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6813","nonce":3911525610}]},"public_addr":"172.21.15.88:6809/3911525610","cluster_addr":"172.21.15.88:6811/3911525610","heartbeat_back_addr":"172.21.15.88:6815/3911525610","heartbeat_front_addr":"172.21.15.88:6813/3911525610","state":["exists","up"]},{"osd":5,"uuid":"5a2a56ab-2af9-4176-ba4d-b2138f0e7c75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6809","nonce":905436397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6811","nonce":905436397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6815","nonce":905436397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6813","nonce":905436397}]},"public_addr":"172.21.15.192:6809/905436397","cluster_addr":"172.21.15.192:6811/905436397","heartbeat_back_addr":"172.21.15.192:6815/905436397","heartbeat_front_addr":"172.21.15.192:6813/905436397","state":["exists","up"]},{"osd":6,"uuid":"03eafdda-278c-443c-b376-677eeb50262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6819","nonce":333894423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6821","nonce":333894423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6825","nonce":333894423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6823","nonce":333894423}]},"public_addr":"172.21.15.57:6819/333894423","cluster_addr":"172.21.15.57:6821/333894423","heartbeat_back_addr":"172.21.15.57:6825/333894423","heartbeat_front_addr":"172.21.15.57:6823/333894423","state":["exists","up"]},{"osd":7,"uuid":"3f01254e-be73-41c8-b979-3b6a7534e2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6817","nonce":3031485551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6819","nonce":3031485551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6823","nonce":3031485551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6821","nonce":3031485551}]},"public_addr":"172.21.15.88:6817/3031485551","cluster_addr":"172.21.15.88:6819/3031485551","heartbeat_back_addr":"172.21.15.88:6823/3031485551","heartbeat_front_addr":"172.21.15.88:6821/3031485551","state":["exists","up"]},{"osd":8,"uuid":"624a150e-f47b-4a76-8785-c818dd65342e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6817","nonce":918056390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6819","nonce":918056390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6823","nonce":918056390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6821","nonce":918056390}]},"public_addr":"172.21.15.192:6817/918056390","cluster_addr":"172.21.15.192:6819/918056390","heartbeat_back_addr":"172.21.15.192:6823/918056390","heartbeat_front_addr":"172.21.15.192:6821/918056390","state":["exists","up"]},{"osd":9,"uuid":"6d9cb69d-ed7b-4009-9ec5-027becc577d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6827","nonce":1773340763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6829","nonce":1773340763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6833","nonce":1773340763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6831","nonce":1773340763}]},"public_addr":"172.21.15.57:6827/1773340763","cluster_addr":"172.21.15.57:6829/1773340763","heartbeat_back_addr":"172.21.15.57:6833/1773340763","heartbeat_front_addr":"172.21.15.57:6831/1773340763","state":["exists","up"]},{"osd":10,"uuid":"d5799219-0979-40b1-a596-acd37f37a72d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6825","nonce":2781975579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6827","nonce":2781975579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6831","nonce":2781975579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6829","nonce":2781975579}]},"public_addr":"172.21.15.88:6825/2781975579","cluster_addr":"172.21.15.88:6827/2781975579","heartbeat_back_addr":"172.21.15.88:6831/2781975579","heartbeat_front_addr":"172.21.15.88:6829/2781975579","state":["exists","up"]},{"osd":11,"uuid":"e77cac6c-107e-41f9-8efd-e1140c4e3bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6825","nonce":2864190031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6827","nonce":2864190031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6831","nonce":2864190031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6829","nonce":2864190031}]},"public_addr":"172.21.15.192:6825/2864190031","cluster_addr":"172.21.15.192:6827/2864190031","heartbeat_back_addr":"172.21.15.192:6831/2864190031","heartbeat_front_addr":"172.21.15.192:6829/2864190031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:53:38.180971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:00.253039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:20.776122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:48.631094+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:09.288231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:29.595802+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:55.089126+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:15.637849+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:35.985101+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:01.544767+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:22.351743+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:43.255157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6801/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/3804015220":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6800/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1582727629":"2023-07-21T18:51:34.614904+0000","172.21.15.57:6800/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/2044626900":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/748511752":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1151100650":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6801/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/3446341201":"2023-07-21T18:52:02.610729+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T18:57:57.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.126+0000 7f90e73ff700 1 -- 172.21.15.57:0/182168369 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f90cc0a11a0 msgr2=0x7f90cc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:57.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.126+0000 7f90e73ff700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f90cc0a11a0 0x7f90cc0a3660 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f90e01017c0 tx=0x7f90dc00b040 comp rx=0 tx=0).stop 2023-07-20T18:57:57.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.126+0000 7f90e73ff700 1 -- 172.21.15.57:0/182168369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e0101960 msgr2=0x7f90e0072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:57.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.126+0000 7f90e73ff700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e0101960 0x7f90e0072b60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009230 tx=0x7f90d0005270 comp rx=0 tx=0).stop 2023-07-20T18:57:57.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.126+0000 7f90e73ff700 1 -- 172.21.15.57:0/182168369 shutdown_connections 2023-07-20T18:57:57.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.126+0000 7f90e73ff700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f90cc0a11a0 0x7f90cc0a3660 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.126+0000 7f90e73ff700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90e0102300 0x7f90e00730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.126+0000 7f90e73ff700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90e0100770 0x7f90e0074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.126+0000 7f90e73ff700 1 --2- 172.21.15.57:0/182168369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e0101960 0x7f90e0072b60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.126+0000 7f90e73ff700 1 -- 172.21.15.57:0/182168369 >> 172.21.15.57:0/182168369 conn(0x7f90e00fbca0 msgr2=0x7f90e0104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:57.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.127+0000 7f90e73ff700 1 -- 172.21.15.57:0/182168369 shutdown_connections 2023-07-20T18:57:57.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.127+0000 7f90e73ff700 1 -- 172.21.15.57:0/182168369 wait complete. 2023-07-20T18:57:57.193 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:57:57.193 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: Saving service mds.all spec with placement smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2023-07-20T18:57:57.193 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:57.193 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:57.194 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:57.194 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:57.194 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:57.194 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:57:57.194 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:57:57.194 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:57.195 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:56 smithi057 ceph-mon[126340]: Deploying daemon mds.l on smithi192 2023-07-20T18:57:57.196 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T18:54:23.215777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T18:57:57.196 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T18:57:57.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:57:57.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: Saving service mds.all spec with placement smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2023-07-20T18:57:57.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:57.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:57.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:57.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:57.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:57.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:57:57.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:57:57.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:57.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:56 smithi192 ceph-mon[129200]: Deploying daemon mds.l on smithi192 2023-07-20T18:57:57.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:57:57.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: Saving service mds.all spec with placement smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2023-07-20T18:57:57.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:57.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:57:57.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:57.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:57:57.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:57.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:57:57.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:57:57.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:57.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:56 smithi088 ceph-mon[128823]: Deploying daemon mds.l on smithi192 2023-07-20T18:57:57.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.353+0000 7f6b4bdff700 1 -- 172.21.15.57:0/1047035270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b44101950 msgr2=0x7f6b44101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:57.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.353+0000 7f6b4bdff700 1 --2- 172.21.15.57:0/1047035270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b44101950 0x7f6b44101db0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6b340099c0 tx=0x7f6b34009cd0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.353+0000 7f6b4bdff700 1 -- 172.21.15.57:0/1047035270 shutdown_connections 2023-07-20T18:57:57.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.353+0000 7f6b4bdff700 1 --2- 172.21.15.57:0/1047035270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b44100760 0x7f6b44100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.353+0000 7f6b4bdff700 1 --2- 172.21.15.57:0/1047035270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b441022f0 0x7f6b44108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.353+0000 7f6b4bdff700 1 --2- 172.21.15.57:0/1047035270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b44101950 0x7f6b44101db0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.353+0000 7f6b4bdff700 1 -- 172.21.15.57:0/1047035270 >> 172.21.15.57:0/1047035270 conn(0x7f6b440fbcb0 msgr2=0x7f6b440fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:57.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.353+0000 7f6b4bdff700 1 -- 172.21.15.57:0/1047035270 shutdown_connections 2023-07-20T18:57:57.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.353+0000 7f6b4bdff700 1 -- 172.21.15.57:0/1047035270 wait complete. 2023-07-20T18:57:57.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.354+0000 7f6b4bdff700 1 Processor -- start 2023-07-20T18:57:57.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b4bdff700 1 -- start start 2023-07-20T18:57:57.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b4bdff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b44100760 0x7f6b44198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:57.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b4bdff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b44101950 0x7f6b44198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:57.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b4bdff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b441022f0 0x7f6b4419fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:57.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b4bdff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b441a02f0 con 0x7f6b44100760 2023-07-20T18:57:57.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b4bdff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b441a0430 con 0x7f6b44101950 2023-07-20T18:57:57.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b4bdff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b441a05c0 con 0x7f6b441022f0 2023-07-20T18:57:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b49b9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b44100760 0x7f6b44198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b4a39c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b441022f0 0x7f6b4419fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b49b9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b44100760 0x7f6b44198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41862/0 (socket says 172.21.15.57:41862) 2023-07-20T18:57:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b49b9b700 1 -- 172.21.15.57:0/2633362706 learned_addr learned my addr 172.21.15.57:0/2633362706 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:57:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b4939a700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b44101950 0x7f6b44198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b49b9b700 1 -- 172.21.15.57:0/2633362706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b441022f0 msgr2=0x7f6b4419fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b49b9b700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b441022f0 0x7f6b4419fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b49b9b700 1 -- 172.21.15.57:0/2633362706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b44101950 msgr2=0x7f6b44198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b49b9b700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b44101950 0x7f6b44198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b49b9b700 1 -- 172.21.15.57:0/2633362706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b340095f0 con 0x7f6b44100760 2023-07-20T18:57:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.355+0000 7f6b4a39c700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b441022f0 0x7f6b4419fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:57:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.356+0000 7f6b49b9b700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b44100760 0x7f6b44198770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000e970 tx=0x7f6b4000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.356+0000 7f6b3affd700 1 -- 172.21.15.57:0/2633362706 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40009790 con 0x7f6b44100760 2023-07-20T18:57:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.356+0000 7f6b4bdff700 1 -- 172.21.15.57:0/2633362706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b441a08a0 con 0x7f6b44100760 2023-07-20T18:57:57.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.356+0000 7f6b3affd700 1 -- 172.21.15.57:0/2633362706 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f6b4000ccc0 con 0x7f6b44100760 2023-07-20T18:57:57.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.356+0000 7f6b3affd700 1 -- 172.21.15.57:0/2633362706 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40003d80 con 0x7f6b44100760 2023-07-20T18:57:57.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.356+0000 7f6b4bdff700 1 -- 172.21.15.57:0/2633362706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b441a0e00 con 0x7f6b44100760 2023-07-20T18:57:57.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.357+0000 7f6b4bdff700 1 -- 172.21.15.57:0/2633362706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4410c100 con 0x7f6b44100760 2023-07-20T18:57:57.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.359+0000 7f6b3affd700 1 -- 172.21.15.57:0/2633362706 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6b4000c7e0 con 0x7f6b44100760 2023-07-20T18:57:57.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.360+0000 7f6b3affd700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6b300a11d0 0x7f6b300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:57.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.361+0000 7f6b3affd700 1 -- 172.21.15.57:0/2633362706 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6b40014070 con 0x7f6b44100760 2023-07-20T18:57:57.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.361+0000 7f6b4939a700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6b300a11d0 0x7f6b300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:57.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.361+0000 7f6b4939a700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6b300a11d0 0x7f6b300a3690 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6b34000c00 tx=0x7f6b34005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:57.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.364+0000 7f6b3affd700 1 -- 172.21.15.57:0/2633362706 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6b40096180 con 0x7f6b44100760 2023-07-20T18:57:57.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.574+0000 7f6b4bdff700 1 -- 172.21.15.57:0/2633362706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b441a1440 con 0x7f6b44100760 2023-07-20T18:57:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.575+0000 7f6b3affd700 1 -- 172.21.15.57:0/2633362706 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b400997a0 con 0x7f6b44100760 2023-07-20T18:57:57.624 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-07-20T18:57:57.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.577+0000 7f6b4bdff700 1 -- 172.21.15.57:0/2633362706 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6b300a11d0 msgr2=0x7f6b300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:57.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.577+0000 7f6b4bdff700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6b300a11d0 0x7f6b300a3690 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6b34000c00 tx=0x7f6b34005fb0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.577+0000 7f6b4bdff700 1 -- 172.21.15.57:0/2633362706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b44100760 msgr2=0x7f6b44198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:57.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.577+0000 7f6b4bdff700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b44100760 0x7f6b44198770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000e970 tx=0x7f6b4000ec80 comp rx=0 tx=0).stop 2023-07-20T18:57:57.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.577+0000 7f6b4bdff700 1 -- 172.21.15.57:0/2633362706 shutdown_connections 2023-07-20T18:57:57.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.577+0000 7f6b4bdff700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6b300a11d0 0x7f6b300a3690 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.577+0000 7f6b4bdff700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b441022f0 0x7f6b4419fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.577+0000 7f6b4bdff700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b44100760 0x7f6b44198770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.577+0000 7f6b4bdff700 1 --2- 172.21.15.57:0/2633362706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b44101950 0x7f6b44198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.577+0000 7f6b4bdff700 1 -- 172.21.15.57:0/2633362706 >> 172.21.15.57:0/2633362706 conn(0x7f6b440fbcb0 msgr2=0x7f6b440fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.578+0000 7f6b4bdff700 1 -- 172.21.15.57:0/2633362706 shutdown_connections 2023-07-20T18:57:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.578+0000 7f6b4bdff700 1 -- 172.21.15.57:0/2633362706 wait complete. 2023-07-20T18:57:57.647 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:57:57.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.803+0000 7f0ed5fff700 1 -- 172.21.15.57:0/2079883940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ed0101960 msgr2=0x7f0ed0101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:57.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.803+0000 7f0ed5fff700 1 --2- 172.21.15.57:0/2079883940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ed0101960 0x7f0ed0101dc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40098f0 tx=0x7f0ec4009c00 comp rx=0 tx=0).stop 2023-07-20T18:57:57.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.803+0000 7f0ed5fff700 1 -- 172.21.15.57:0/2079883940 shutdown_connections 2023-07-20T18:57:57.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.803+0000 7f0ed5fff700 1 --2- 172.21.15.57:0/2079883940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ed0102300 0x7f0ed0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.803+0000 7f0ed5fff700 1 --2- 172.21.15.57:0/2079883940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ed0100770 0x7f0ed0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.803+0000 7f0ed5fff700 1 --2- 172.21.15.57:0/2079883940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ed0101960 0x7f0ed0101dc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.803+0000 7f0ed5fff700 1 -- 172.21.15.57:0/2079883940 >> 172.21.15.57:0/2079883940 conn(0x7f0ed00fbca0 msgr2=0x7f0ed00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:57.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.803+0000 7f0ed5fff700 1 -- 172.21.15.57:0/2079883940 shutdown_connections 2023-07-20T18:57:57.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.803+0000 7f0ed5fff700 1 -- 172.21.15.57:0/2079883940 wait complete. 2023-07-20T18:57:57.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.804+0000 7f0ed5fff700 1 Processor -- start 2023-07-20T18:57:57.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.804+0000 7f0ed5fff700 1 -- start start 2023-07-20T18:57:57.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.804+0000 7f0ed5fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ed0100770 0x7f0ed01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:57.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.804+0000 7f0ed5fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ed0101960 0x7f0ed0198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:57.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.804+0000 7f0ed5fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ed0102300 0x7f0ed019fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:57.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.804+0000 7f0ed5fff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed01a0330 con 0x7f0ed0101960 2023-07-20T18:57:57.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.804+0000 7f0ed5fff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed01a0470 con 0x7f0ed0100770 2023-07-20T18:57:57.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.804+0000 7f0ed5fff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed01a0600 con 0x7f0ed0102300 2023-07-20T18:57:57.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.804+0000 7f0eceffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ed0101960 0x7f0ed0198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:57.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.804+0000 7f0ecf7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ed0100770 0x7f0ed01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:57.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0ecf7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ed0100770 0x7f0ed01987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:51394/0 (socket says 172.21.15.57:51394) 2023-07-20T18:57:57.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0ecf7fe700 1 -- 172.21.15.57:0/120091455 learned_addr learned my addr 172.21.15.57:0/120091455 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:57:57.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0ecffff700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ed0102300 0x7f0ed019fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:57.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0ecf7fe700 1 -- 172.21.15.57:0/120091455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ed0102300 msgr2=0x7f0ed019fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:57.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0ecf7fe700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ed0102300 0x7f0ed019fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0ecf7fe700 1 -- 172.21.15.57:0/120091455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ed0101960 msgr2=0x7f0ed0198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:57.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0ecf7fe700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ed0101960 0x7f0ed0198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:57.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0ecf7fe700 1 -- 172.21.15.57:0/120091455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec4009520 con 0x7f0ed0100770 2023-07-20T18:57:57.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0eceffd700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ed0101960 0x7f0ed0198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:57:57.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0ecf7fe700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ed0100770 0x7f0ed01987b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000e9f0 tx=0x7f0ec000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:57.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0eccff9700 1 -- 172.21.15.57:0/120091455 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0004220 con 0x7f0ed0100770 2023-07-20T18:57:57.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0eccff9700 1 -- 172.21.15.57:0/120091455 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f0ec0004380 con 0x7f0ed0100770 2023-07-20T18:57:57.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.805+0000 7f0ed5fff700 1 -- 172.21.15.57:0/120091455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed01a08e0 con 0x7f0ed0100770 2023-07-20T18:57:57.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.806+0000 7f0eccff9700 1 -- 172.21.15.57:0/120091455 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0003c50 con 0x7f0ed0100770 2023-07-20T18:57:57.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.806+0000 7f0ed5fff700 1 -- 172.21.15.57:0/120091455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed01a0e40 con 0x7f0ed0100770 2023-07-20T18:57:57.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.806+0000 7f0ed5fff700 1 -- 172.21.15.57:0/120091455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed0192f00 con 0x7f0ed0100770 2023-07-20T18:57:57.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.809+0000 7f0eccff9700 1 -- 172.21.15.57:0/120091455 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0ec000c7c0 con 0x7f0ed0100770 2023-07-20T18:57:57.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.810+0000 7f0eccff9700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0eb80a1220 0x7f0eb80a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:57:57.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.810+0000 7f0eccff9700 1 -- 172.21.15.57:0/120091455 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0ec0014070 con 0x7f0ed0100770 2023-07-20T18:57:57.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.810+0000 7f0eceffd700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0eb80a1220 0x7f0eb80a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:57:57.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.811+0000 7f0eceffd700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0eb80a1220 0x7f0eb80a36e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400fbc0 tx=0x7f0ec401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:57:57.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:57.812+0000 7f0eccff9700 1 -- 172.21.15.57:0/120091455 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ec0096140 con 0x7f0ed0100770 2023-07-20T18:57:57.908 INFO:journalctl@ceph.mds.l.smithi192.stdout:Jul 20 18:57:57 smithi192 systemd[1]: Starting Ceph mds.l for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:57:58.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.066+0000 7f0ed5fff700 1 -- 172.21.15.57:0/120091455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ed002c970 con 0x7f0ed0100770 2023-07-20T18:57:58.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.067+0000 7f0eccff9700 1 -- 172.21.15.57:0/120091455 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f0ec0099760 con 0x7f0ed0100770 2023-07-20T18:57:58.116 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:57:58.116 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-20T18:57:58.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 -- 172.21.15.57:0/120091455 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0eb80a1220 msgr2=0x7f0eb80a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:58.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0eb80a1220 0x7f0eb80a36e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400fbc0 tx=0x7f0ec401a040 comp rx=0 tx=0).stop 2023-07-20T18:57:58.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 -- 172.21.15.57:0/120091455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ed0100770 msgr2=0x7f0ed01987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:57:58.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ed0100770 0x7f0ed01987b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000e9f0 tx=0x7f0ec000ed00 comp rx=0 tx=0).stop 2023-07-20T18:57:58.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 -- 172.21.15.57:0/120091455 shutdown_connections 2023-07-20T18:57:58.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f0eb80a1220 0x7f0eb80a36e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:58.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ed0102300 0x7f0ed019fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:58.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ed0100770 0x7f0ed01987b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:58.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 --2- 172.21.15.57:0/120091455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ed0101960 0x7f0ed0198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:57:58.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 -- 172.21.15.57:0/120091455 >> 172.21.15.57:0/120091455 conn(0x7f0ed00fbca0 msgr2=0x7f0ed0107270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:57:58.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 -- 172.21.15.57:0/120091455 shutdown_connections 2023-07-20T18:57:58.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:57:58.069+0000 7f0ed5fff700 1 -- 172.21.15.57:0/120091455 wait complete. 2023-07-20T18:57:58.120 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 1 2023-07-20T18:57:58.167 INFO:journalctl@ceph.mds.l.smithi192.stdout:Jul 20 18:57:57 smithi192 podman[154395]: 2023-07-20T18:57:58.167 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:57 smithi192 ceph-mon[129200]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:58.167 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:57 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/182168369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:57:58.168 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:57 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2633362706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:57:58.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:57 smithi088 ceph-mon[128823]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:58.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:57 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/182168369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:57:58.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:57 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2633362706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:57:58.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:57 smithi057 ceph-mon[126340]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:57:58.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:57 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/182168369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:57:58.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:57 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2633362706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:57:58.589 INFO:journalctl@ceph.mds.l.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mds[154436]: -- 172.21.15.192:0/1157089306 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55912e91d1e0 con 0x55912e8f2800 2023-07-20T18:57:58.589 INFO:journalctl@ceph.mds.l.smithi192.stdout:Jul 20 18:57:58 smithi192 bash[154395]: b4acca1e1b1457d4166e532af20eb7232a42f982d19b93abd05e6e79029b623e 2023-07-20T18:57:58.589 INFO:journalctl@ceph.mds.l.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-l[154411]: starting mds.l at 2023-07-20T18:57:58.589 INFO:journalctl@ceph.mds.l.smithi192.stdout:Jul 20 18:57:58 smithi192 systemd[1]: Started Ceph mds.l for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:57:59.161 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:58 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/120091455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:57:59.162 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:58 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.192:6832/433531829,v1:172.21.15.192:6833/433531829] up:boot 2023-07-20T18:57:59.162 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:58 smithi088 ceph-mon[128823]: fsmap 1 up:standby 2023-07-20T18:57:59.162 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T18:57:59.162 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:59.162 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:59.163 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:59.163 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:57:59.163 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:57:59.163 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:58 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:59.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/120091455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:57:59.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.192:6832/433531829,v1:172.21.15.192:6833/433531829] up:boot 2023-07-20T18:57:59.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mon[129200]: fsmap 1 up:standby 2023-07-20T18:57:59.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T18:57:59.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:59.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:59.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:59.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:57:59.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:57:59.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:58 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:57:59.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:58 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/120091455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:57:59.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:58 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.192:6832/433531829,v1:172.21.15.192:6833/433531829] up:boot 2023-07-20T18:57:59.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:58 smithi057 ceph-mon[126340]: fsmap 1 up:standby 2023-07-20T18:57:59.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T18:57:59.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:59.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:59.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:57:59.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:57:59.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:57:59.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:58 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:00.022 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 20 18:57:59 smithi088 systemd[1]: Starting Ceph mds.k for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:00.138 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:00.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:59 smithi192 ceph-mon[129200]: Deploying daemon mds.k on smithi088 2023-07-20T18:58:00.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:57:59 smithi192 ceph-mon[129200]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.310+0000 7f52d65ff700 1 -- 172.21.15.57:0/153731683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0073190 msgr2=0x7f52d00735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.310+0000 7f52d65ff700 1 --2- 172.21.15.57:0/153731683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0073190 0x7f52d00735f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52c4009990 tx=0x7f52c4009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.311+0000 7f52d65ff700 1 -- 172.21.15.57:0/153731683 shutdown_connections 2023-07-20T18:58:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.311+0000 7f52d65ff700 1 --2- 172.21.15.57:0/153731683 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d0074770 0x7f52d0072bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.311+0000 7f52d65ff700 1 --2- 172.21.15.57:0/153731683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0108620 0x7f52d010aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.311+0000 7f52d65ff700 1 --2- 172.21.15.57:0/153731683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0073190 0x7f52d00735f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.311+0000 7f52d65ff700 1 -- 172.21.15.57:0/153731683 >> 172.21.15.57:0/153731683 conn(0x7f52d00fbca0 msgr2=0x7f52d00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.311+0000 7f52d65ff700 1 -- 172.21.15.57:0/153731683 shutdown_connections 2023-07-20T18:58:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.311+0000 7f52d65ff700 1 -- 172.21.15.57:0/153731683 wait complete. 2023-07-20T18:58:00.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.312+0000 7f52d65ff700 1 Processor -- start 2023-07-20T18:58:00.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.312+0000 7f52d65ff700 1 -- start start 2023-07-20T18:58:00.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.312+0000 7f52d65ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d0073190 0x7f52d0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:00.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52d65ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0074770 0x7f52d0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:00.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52d65ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0108620 0x7f52d019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:00.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52d65ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d01a02a0 con 0x7f52d0108620 2023-07-20T18:58:00.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52d65ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d01a03e0 con 0x7f52d0074770 2023-07-20T18:58:00.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52d65ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d01a0570 con 0x7f52d0073190 2023-07-20T18:58:00.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52cf7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0074770 0x7f52d0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:00.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52d4b9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0108620 0x7f52d019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:00.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52cf7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0074770 0x7f52d0198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:51418/0 (socket says 172.21.15.57:51418) 2023-07-20T18:58:00.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52cf7fe700 1 -- 172.21.15.57:0/234493181 learned_addr learned my addr 172.21.15.57:0/234493181 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:00.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52cffff700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d0073190 0x7f52d0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:00.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52cf7fe700 1 -- 172.21.15.57:0/234493181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d0073190 msgr2=0x7f52d0198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:00.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52cf7fe700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d0073190 0x7f52d0198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:00.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52cf7fe700 1 -- 172.21.15.57:0/234493181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0108620 msgr2=0x7f52d019fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:00.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52cf7fe700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0108620 0x7f52d019fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:00.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52cf7fe700 1 -- 172.21.15.57:0/234493181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c40095f0 con 0x7f52d0074770 2023-07-20T18:58:00.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.313+0000 7f52d4b9c700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0108620 0x7f52d019fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:00.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.314+0000 7f52cf7fe700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0074770 0x7f52d0198c60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f52c4004730 tx=0x7f52c4004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:00.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.314+0000 7f52cd7fa700 1 -- 172.21.15.57:0/234493181 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c401d070 con 0x7f52d0074770 2023-07-20T18:58:00.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.314+0000 7f52d65ff700 1 -- 172.21.15.57:0/234493181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d01a07f0 con 0x7f52d0074770 2023-07-20T18:58:00.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.314+0000 7f52cd7fa700 1 -- 172.21.15.57:0/234493181 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f52c4004cb0 con 0x7f52d0074770 2023-07-20T18:58:00.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.314+0000 7f52cd7fa700 1 -- 172.21.15.57:0/234493181 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c4005460 con 0x7f52d0074770 2023-07-20T18:58:00.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.314+0000 7f52d65ff700 1 -- 172.21.15.57:0/234493181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d01a0bb0 con 0x7f52d0074770 2023-07-20T18:58:00.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.315+0000 7f52d65ff700 1 -- 172.21.15.57:0/234493181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d0107430 con 0x7f52d0074770 2023-07-20T18:58:00.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.318+0000 7f52cd7fa700 1 -- 172.21.15.57:0/234493181 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f52c400fb40 con 0x7f52d0074770 2023-07-20T18:58:00.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.321+0000 7f52cd7fa700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f52b80a11a0 0x7f52b80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:00.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.321+0000 7f52cd7fa700 1 -- 172.21.15.57:0/234493181 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f52c40cd6e0 con 0x7f52d0074770 2023-07-20T18:58:00.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.321+0000 7f52cffff700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f52b80a11a0 0x7f52b80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:00.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.321+0000 7f52cd7fa700 1 -- 172.21.15.57:0/234493181 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f52c40cdb20 con 0x7f52d0074770 2023-07-20T18:58:00.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.321+0000 7f52cffff700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f52b80a11a0 0x7f52b80a3660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009c60 tx=0x7f52c0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:00.381 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:59 smithi088 ceph-mon[128823]: Deploying daemon mds.k on smithi088 2023-07-20T18:58:00.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:57:59 smithi088 ceph-mon[128823]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:00.382 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 20 18:57:59 smithi088 podman[153125]: 2023-07-20T18:58:00.382 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 20 18:58:00 smithi088 ceph-mds[153165]: -- 172.21.15.88:0/81039882 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559658c691e0 con 0x559658c3f000 2023-07-20T18:58:00.382 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 20 18:58:00 smithi088 bash[153125]: 7830deb100726196ebad6f551c1f538538faf2393764cc96e95821ed4e6535eb 2023-07-20T18:58:00.382 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 20 18:58:00 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-k[153141]: starting mds.k at 2023-07-20T18:58:00.383 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 20 18:58:00 smithi088 systemd[1]: Started Ceph mds.k for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:00.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:59 smithi057 ceph-mon[126340]: Deploying daemon mds.k on smithi088 2023-07-20T18:58:00.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:57:59 smithi057 ceph-mon[126340]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:00.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.590+0000 7f52d65ff700 1 -- 172.21.15.57:0/234493181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52d002c970 con 0x7f52d0074770 2023-07-20T18:58:00.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.591+0000 7f52cd7fa700 1 -- 172.21.15.57:0/234493181 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1257 (secure 0 0 0) 0x7f52c40972c0 con 0x7f52d0074770 2023-07-20T18:58:00.640 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:00.640 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T18:58:00.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 -- 172.21.15.57:0/234493181 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f52b80a11a0 msgr2=0x7f52b80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:00.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f52b80a11a0 0x7f52b80a3660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009c60 tx=0x7f52c0009260 comp rx=0 tx=0).stop 2023-07-20T18:58:00.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 -- 172.21.15.57:0/234493181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0074770 msgr2=0x7f52d0198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:00.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0074770 0x7f52d0198c60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f52c4004730 tx=0x7f52c4004760 comp rx=0 tx=0).stop 2023-07-20T18:58:00.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 -- 172.21.15.57:0/234493181 shutdown_connections 2023-07-20T18:58:00.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f52b80a11a0 0x7f52b80a3660 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:00.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d0073190 0x7f52d0198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:00.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0108620 0x7f52d019fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:00.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 --2- 172.21.15.57:0/234493181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0074770 0x7f52d0198c60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:00.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 -- 172.21.15.57:0/234493181 >> 172.21.15.57:0/234493181 conn(0x7f52d00fbca0 msgr2=0x7f52d0104570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:00.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 -- 172.21.15.57:0/234493181 shutdown_connections 2023-07-20T18:58:00.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:00.594+0000 7f52d65ff700 1 -- 172.21.15.57:0/234493181 wait complete. 2023-07-20T18:58:00.645 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 2 2023-07-20T18:58:01.544 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:01.544 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:01.545 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:01.545 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:01.545 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:01.545 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:01.545 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: Deploying daemon mds.j on smithi057 2023-07-20T18:58:01.546 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/234493181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:01.546 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:01.546 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.88:6832/749062861,v1:172.21.15.88:6833/749062861] up:boot 2023-07-20T18:58:01.546 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: fsmap 2 up:standby 2023-07-20T18:58:01.546 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:01 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T18:58:01.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:01.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:01.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: Deploying daemon mds.j on smithi057 2023-07-20T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/234493181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.88:6832/749062861,v1:172.21.15.88:6833/749062861] up:boot 2023-07-20T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: fsmap 2 up:standby 2023-07-20T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:01 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T18:58:01.632 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:01.632 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:01.632 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:01.632 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:01.632 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:01.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:01.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: Deploying daemon mds.j on smithi057 2023-07-20T18:58:01.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/234493181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:01.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:01.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.88:6832/749062861,v1:172.21.15.88:6833/749062861] up:boot 2023-07-20T18:58:01.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: fsmap 2 up:standby 2023-07-20T18:58:01.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:01 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T18:58:02.449 INFO:journalctl@ceph.mds.j.smithi057.stdout:Jul 20 18:58:02 smithi057 systemd[1]: Starting Ceph mds.j for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:02.664 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:02.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.846+0000 7f44883ff700 1 -- 172.21.15.57:0/3980432805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4480073190 msgr2=0x7f44800735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:02.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.846+0000 7f44883ff700 1 --2- 172.21.15.57:0/3980432805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4480073190 0x7f44800735f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4470009990 tx=0x7f4470009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.846+0000 7f44883ff700 1 -- 172.21.15.57:0/3980432805 shutdown_connections 2023-07-20T18:58:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.846+0000 7f44883ff700 1 --2- 172.21.15.57:0/3980432805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4480074770 0x7f4480072bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.846+0000 7f44883ff700 1 --2- 172.21.15.57:0/3980432805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4480108620 0x7f448010aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.846+0000 7f44883ff700 1 --2- 172.21.15.57:0/3980432805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4480073190 0x7f44800735f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.846+0000 7f44883ff700 1 -- 172.21.15.57:0/3980432805 >> 172.21.15.57:0/3980432805 conn(0x7f44800fbca0 msgr2=0x7f44800fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.846+0000 7f44883ff700 1 -- 172.21.15.57:0/3980432805 shutdown_connections 2023-07-20T18:58:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.846+0000 7f44883ff700 1 -- 172.21.15.57:0/3980432805 wait complete. 2023-07-20T18:58:02.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.847+0000 7f44883ff700 1 Processor -- start 2023-07-20T18:58:02.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.847+0000 7f44883ff700 1 -- start start 2023-07-20T18:58:02.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.847+0000 7f44883ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4480073190 0x7f44801986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:02.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.847+0000 7f44883ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4480074770 0x7f4480198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:02.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f44883ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4480108620 0x7f448019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:02.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f44883ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44801a0250 con 0x7f4480074770 2023-07-20T18:58:02.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f44883ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44801a0390 con 0x7f4480108620 2023-07-20T18:58:02.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f44883ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44801a0520 con 0x7f4480073190 2023-07-20T18:58:02.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448599a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4480074770 0x7f4480198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:02.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448619b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4480073190 0x7f44801986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:02.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448599a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4480074770 0x7f4480198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41932/0 (socket says 172.21.15.57:41932) 2023-07-20T18:58:02.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448599a700 1 -- 172.21.15.57:0/1688021631 learned_addr learned my addr 172.21.15.57:0/1688021631 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:02.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448699c700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4480108620 0x7f448019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:02.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448599a700 1 -- 172.21.15.57:0/1688021631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4480073190 msgr2=0x7f44801986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:02.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448599a700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4480073190 0x7f44801986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:02.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448599a700 1 -- 172.21.15.57:0/1688021631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4480108620 msgr2=0x7f448019fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:02.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448599a700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4480108620 0x7f448019fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:02.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448599a700 1 -- 172.21.15.57:0/1688021631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44700095f0 con 0x7f4480074770 2023-07-20T18:58:02.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448699c700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4480108620 0x7f448019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:02.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.848+0000 7f448599a700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4480074770 0x7f4480198c10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f447000b780 tx=0x7f447000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:02.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.849+0000 7f44777fe700 1 -- 172.21.15.57:0/1688021631 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447001d070 con 0x7f4480074770 2023-07-20T18:58:02.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.849+0000 7f44883ff700 1 -- 172.21.15.57:0/1688021631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44801a07a0 con 0x7f4480074770 2023-07-20T18:58:02.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.849+0000 7f44777fe700 1 -- 172.21.15.57:0/1688021631 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f447000be40 con 0x7f4480074770 2023-07-20T18:58:02.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.849+0000 7f44777fe700 1 -- 172.21.15.57:0/1688021631 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44700173f0 con 0x7f4480074770 2023-07-20T18:58:02.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.849+0000 7f44883ff700 1 -- 172.21.15.57:0/1688021631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44801a0c20 con 0x7f4480074770 2023-07-20T18:58:02.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.850+0000 7f44883ff700 1 -- 172.21.15.57:0/1688021631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4480106d20 con 0x7f4480074770 2023-07-20T18:58:02.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.852+0000 7f44777fe700 1 -- 172.21.15.57:0/1688021631 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4470022470 con 0x7f4480074770 2023-07-20T18:58:02.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.853+0000 7f44777fe700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f446c0a11a0 0x7f446c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:02.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.853+0000 7f44777fe700 1 -- 172.21.15.57:0/1688021631 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f44700cd800 con 0x7f4480074770 2023-07-20T18:58:02.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.854+0000 7f448619b700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f446c0a11a0 0x7f446c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:02.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.854+0000 7f448619b700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f446c0a11a0 0x7f446c0a3660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f447c009c40 tx=0x7f447c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:02.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:02.857+0000 7f44777fe700 1 -- 172.21.15.57:0/1688021631 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f447005c780 con 0x7f4480074770 2023-07-20T18:58:03.096 INFO:journalctl@ceph.mds.j.smithi057.stdout:Jul 20 18:58:02 smithi057 podman[159786]: 2023-07-20T18:58:03.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.164+0000 7f44883ff700 1 -- 172.21.15.57:0/1688021631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4480061380 con 0x7f4480074770 2023-07-20T18:58:03.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.165+0000 7f44777fe700 1 -- 172.21.15.57:0/1688021631 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2011 (secure 0 0 0) 0x7f44700973e0 con 0x7f4480074770 2023-07-20T18:58:03.215 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:03.215 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-20T18:58:03.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 -- 172.21.15.57:0/1688021631 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f446c0a11a0 msgr2=0x7f446c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:03.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f446c0a11a0 0x7f446c0a3660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f447c009c40 tx=0x7f447c009260 comp rx=0 tx=0).stop 2023-07-20T18:58:03.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 -- 172.21.15.57:0/1688021631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4480074770 msgr2=0x7f4480198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:03.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4480074770 0x7f4480198c10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f447000b780 tx=0x7f447000f690 comp rx=0 tx=0).stop 2023-07-20T18:58:03.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 -- 172.21.15.57:0/1688021631 shutdown_connections 2023-07-20T18:58:03.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f446c0a11a0 0x7f446c0a3660 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:03.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4480073190 0x7f44801986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:03.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4480108620 0x7f448019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:03.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 --2- 172.21.15.57:0/1688021631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4480074770 0x7f4480198c10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:03.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 -- 172.21.15.57:0/1688021631 >> 172.21.15.57:0/1688021631 conn(0x7f44800fbca0 msgr2=0x7f44800fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:03.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 -- 172.21.15.57:0/1688021631 shutdown_connections 2023-07-20T18:58:03.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:03.168+0000 7f44883ff700 1 -- 172.21.15.57:0/1688021631 wait complete. 2023-07-20T18:58:03.220 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 3 2023-07-20T18:58:03.484 INFO:journalctl@ceph.mds.j.smithi057.stdout:Jul 20 18:58:03 smithi057 bash[159786]: b1f47bdf7cca48338c74219355f4a08b5c694c6979f21331e558b2ecbd3ad6df 2023-07-20T18:58:03.485 INFO:journalctl@ceph.mds.j.smithi057.stdout:Jul 20 18:58:03 smithi057 systemd[1]: Started Ceph mds.j for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:03.485 INFO:journalctl@ceph.mds.j.smithi057.stdout:Jul 20 18:58:03 smithi057 ceph-mds[159846]: -- 172.21.15.57:0/3255081419 <== mon.0 v2:172.21.15.57:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d15fd2f1e0 con 0x55d15fd04800 2023-07-20T18:58:03.485 INFO:journalctl@ceph.mds.j.smithi057.stdout:Jul 20 18:58:03 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-j[159822]: starting mds.j at 2023-07-20T18:58:04.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:03 smithi192 ceph-mon[129200]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:04.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:03 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1688021631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:04.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:04.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:04.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:04.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:04.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:04.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:04.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:03 smithi088 ceph-mon[128823]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:04.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:03 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1688021631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:04.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:04.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:04.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:04.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:04.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:04.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:04.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:03 smithi057 ceph-mon[126340]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:04.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:03 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1688021631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:04.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:04.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:04.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:04.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:04.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:04.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:04.858 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:04 smithi057 ceph-mon[126340]: Deploying daemon mds.d on smithi057 2023-07-20T18:58:04.858 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:04 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.57:6834/4001514318,v1:172.21.15.57:6835/4001514318] up:boot 2023-07-20T18:58:04.859 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:04 smithi057 ceph-mon[126340]: fsmap 3 up:standby 2023-07-20T18:58:04.859 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:04 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T18:58:05.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:04 smithi192 ceph-mon[129200]: Deploying daemon mds.d on smithi057 2023-07-20T18:58:05.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:04 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.57:6834/4001514318,v1:172.21.15.57:6835/4001514318] up:boot 2023-07-20T18:58:05.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:04 smithi192 ceph-mon[129200]: fsmap 3 up:standby 2023-07-20T18:58:05.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:04 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T18:58:05.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:04 smithi088 ceph-mon[128823]: Deploying daemon mds.d on smithi057 2023-07-20T18:58:05.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:04 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.57:6834/4001514318,v1:172.21.15.57:6835/4001514318] up:boot 2023-07-20T18:58:05.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:04 smithi088 ceph-mon[128823]: fsmap 3 up:standby 2023-07-20T18:58:05.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:04 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T18:58:05.240 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:05.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.568+0000 7f9018dff700 1 -- 172.21.15.57:0/1958996521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9014102310 msgr2=0x7f9014108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:05.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.568+0000 7f9018dff700 1 --2- 172.21.15.57:0/1958996521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9014102310 0x7f9014108bc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f900c00b0d0 tx=0x7f900c00b3e0 comp rx=0 tx=0).stop 2023-07-20T18:58:05.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.568+0000 7f9018dff700 1 -- 172.21.15.57:0/1958996521 shutdown_connections 2023-07-20T18:58:05.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.568+0000 7f9018dff700 1 --2- 172.21.15.57:0/1958996521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9014102310 0x7f9014108bc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:05.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.568+0000 7f9018dff700 1 --2- 172.21.15.57:0/1958996521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9014100780 0x7f9014100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:05.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.568+0000 7f9018dff700 1 --2- 172.21.15.57:0/1958996521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9014101970 0x7f9014101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:05.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.568+0000 7f9018dff700 1 -- 172.21.15.57:0/1958996521 >> 172.21.15.57:0/1958996521 conn(0x7f90140fbcb0 msgr2=0x7f90140fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:05.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.568+0000 7f9018dff700 1 -- 172.21.15.57:0/1958996521 shutdown_connections 2023-07-20T18:58:05.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.568+0000 7f9018dff700 1 -- 172.21.15.57:0/1958996521 wait complete. 2023-07-20T18:58:05.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.569+0000 7f9018dff700 1 Processor -- start 2023-07-20T18:58:05.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.570+0000 7f9018dff700 1 -- start start 2023-07-20T18:58:05.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.570+0000 7f9018dff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9014100780 0x7f9014074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.570+0000 7f9018dff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9014101970 0x7f9014072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.570+0000 7f9018dff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9014102310 0x7f90140730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.570+0000 7f9018dff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9014073840 con 0x7f9014101970 2023-07-20T18:58:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.570+0000 7f9018dff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90140739b0 con 0x7f9014100780 2023-07-20T18:58:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.570+0000 7f9018dff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90141a0600 con 0x7f9014102310 2023-07-20T18:58:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.570+0000 7f901259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9014100780 0x7f9014074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.570+0000 7f901259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9014100780 0x7f9014074520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:35006/0 (socket says 172.21.15.57:35006) 2023-07-20T18:58:05.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.570+0000 7f901259c700 1 -- 172.21.15.57:0/1826820680 learned_addr learned my addr 172.21.15.57:0/1826820680 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:05.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f9012d9d700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9014102310 0x7f90140730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:05.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f9011d9b700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9014101970 0x7f9014072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:05.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f901259c700 1 -- 172.21.15.57:0/1826820680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9014102310 msgr2=0x7f90140730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:05.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f901259c700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9014102310 0x7f90140730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:05.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f901259c700 1 -- 172.21.15.57:0/1826820680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9014101970 msgr2=0x7f9014072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:05.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f901259c700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9014101970 0x7f9014072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:05.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f901259c700 1 -- 172.21.15.57:0/1826820680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c009d00 con 0x7f9014100780 2023-07-20T18:58:05.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f901259c700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9014100780 0x7f9014074520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f900400e970 tx=0x7f900400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:05.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f90037fe700 1 -- 172.21.15.57:0/1826820680 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004009790 con 0x7f9014100780 2023-07-20T18:58:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f9018dff700 1 -- 172.21.15.57:0/1826820680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90141a08e0 con 0x7f9014100780 2023-07-20T18:58:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f90037fe700 1 -- 172.21.15.57:0/1826820680 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f900400eec0 con 0x7f9014100780 2023-07-20T18:58:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.571+0000 7f90037fe700 1 -- 172.21.15.57:0/1826820680 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004003d40 con 0x7f9014100780 2023-07-20T18:58:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.572+0000 7f9018dff700 1 -- 172.21.15.57:0/1826820680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90141a0df0 con 0x7f9014100780 2023-07-20T18:58:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.573+0000 7f9018dff700 1 -- 172.21.15.57:0/1826820680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901410c0e0 con 0x7f9014100780 2023-07-20T18:58:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.575+0000 7f90037fe700 1 -- 172.21.15.57:0/1826820680 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9004003fe0 con 0x7f9014100780 2023-07-20T18:58:05.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.576+0000 7f90037fe700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8ffc0a11a0 0x7f8ffc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:05.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.576+0000 7f90037fe700 1 -- 172.21.15.57:0/1826820680 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9004014070 con 0x7f9014100780 2023-07-20T18:58:05.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.576+0000 7f9011d9b700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8ffc0a11a0 0x7f8ffc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:05.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.577+0000 7f9011d9b700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8ffc0a11a0 0x7f8ffc0a3660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9008009bf0 tx=0x7f9008009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:05.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.578+0000 7f90037fe700 1 -- 172.21.15.57:0/1826820680 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9004096440 con 0x7f9014100780 2023-07-20T18:58:05.810 INFO:journalctl@ceph.mds.d.smithi057.stdout:Jul 20 18:58:05 smithi057 systemd[1]: Starting Ceph mds.d for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:05.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.868+0000 7f9018dff700 1 -- 172.21.15.57:0/1826820680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9014107570 con 0x7f9014100780 2023-07-20T18:58:05.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.869+0000 7f90037fe700 1 -- 172.21.15.57:0/1826820680 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2768 (secure 0 0 0) 0x7f9004099a60 con 0x7f9014100780 2023-07-20T18:58:05.919 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:05.919 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-20T18:58:05.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 -- 172.21.15.57:0/1826820680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8ffc0a11a0 msgr2=0x7f8ffc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:05.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8ffc0a11a0 0x7f8ffc0a3660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9008009bf0 tx=0x7f9008009260 comp rx=0 tx=0).stop 2023-07-20T18:58:05.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 -- 172.21.15.57:0/1826820680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9014100780 msgr2=0x7f9014074520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:05.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9014100780 0x7f9014074520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f900400e970 tx=0x7f900400ec80 comp rx=0 tx=0).stop 2023-07-20T18:58:05.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 -- 172.21.15.57:0/1826820680 shutdown_connections 2023-07-20T18:58:05.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f8ffc0a11a0 0x7f8ffc0a3660 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:05.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9014102310 0x7f90140730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:05.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9014100780 0x7f9014074520 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:05.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 --2- 172.21.15.57:0/1826820680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9014101970 0x7f9014072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:05.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 -- 172.21.15.57:0/1826820680 >> 172.21.15.57:0/1826820680 conn(0x7f90140fbcb0 msgr2=0x7f90140fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:05.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 -- 172.21.15.57:0/1826820680 shutdown_connections 2023-07-20T18:58:05.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:05.872+0000 7f9018dff700 1 -- 172.21.15.57:0/1826820680 wait complete. 2023-07-20T18:58:05.923 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 4 2023-07-20T18:58:06.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:05 smithi192 ceph-mon[129200]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:06.120 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:05 smithi057 ceph-mon[126340]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:06.121 INFO:journalctl@ceph.mds.d.smithi057.stdout:Jul 20 18:58:05 smithi057 podman[160198]: 2023-07-20T18:58:06.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:05 smithi088 ceph-mon[128823]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:06.484 INFO:journalctl@ceph.mds.d.smithi057.stdout:Jul 20 18:58:06 smithi057 bash[160198]: 71e51e086e26c02673cbaf11dfa05cfe322c153ec6828c691a2b6748bae3adb8 2023-07-20T18:58:06.484 INFO:journalctl@ceph.mds.d.smithi057.stdout:Jul 20 18:58:06 smithi057 ceph-mds[160240]: -- 172.21.15.57:0/2312903767 <== mon.0 v2:172.21.15.57:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5648736f91e0 con 0x5648736cf000 2023-07-20T18:58:06.485 INFO:journalctl@ceph.mds.d.smithi057.stdout:Jul 20 18:58:06 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-d[160216]: starting mds.d at 2023-07-20T18:58:06.485 INFO:journalctl@ceph.mds.d.smithi057.stdout:Jul 20 18:58:06 smithi057 systemd[1]: Started Ceph mds.d for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:07.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:06 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1826820680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:07.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:07.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:07.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:07.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:07.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:07.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:06 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:07.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:06 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1826820680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:07.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:07.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:07.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:07.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:07.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:07.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:06 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:07.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:06 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1826820680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:07.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:07.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:07.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:07.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:07.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:07.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:06 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:07.941 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:08.027 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:07 smithi088 ceph-mon[128823]: Deploying daemon mds.e on smithi088 2023-07-20T18:58:08.028 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:07 smithi088 ceph-mon[128823]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:08.028 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:07 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.57:6836/3251478878,v1:172.21.15.57:6837/3251478878] up:boot 2023-07-20T18:58:08.028 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:07 smithi088 ceph-mon[128823]: fsmap 4 up:standby 2023-07-20T18:58:08.028 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:07 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T18:58:08.028 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 20 18:58:07 smithi088 systemd[1]: Starting Ceph mds.e for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:08.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:07 smithi192 ceph-mon[129200]: Deploying daemon mds.e on smithi088 2023-07-20T18:58:08.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:07 smithi192 ceph-mon[129200]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:08.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:07 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.57:6836/3251478878,v1:172.21.15.57:6837/3251478878] up:boot 2023-07-20T18:58:08.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:07 smithi192 ceph-mon[129200]: fsmap 4 up:standby 2023-07-20T18:58:08.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:07 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T18:58:08.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.111+0000 7f7e451ff700 1 -- 172.21.15.57:0/1137968963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e40100770 msgr2=0x7f7e40100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:08.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.111+0000 7f7e451ff700 1 --2- 172.21.15.57:0/1137968963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e40100770 0x7f7e40100b90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7e300099c0 tx=0x7f7e30009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.111+0000 7f7e451ff700 1 -- 172.21.15.57:0/1137968963 shutdown_connections 2023-07-20T18:58:08.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.111+0000 7f7e451ff700 1 --2- 172.21.15.57:0/1137968963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e40102300 0x7f7e40108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.111+0000 7f7e451ff700 1 --2- 172.21.15.57:0/1137968963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e40100770 0x7f7e40100b90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.111+0000 7f7e451ff700 1 --2- 172.21.15.57:0/1137968963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e40101960 0x7f7e40101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.111+0000 7f7e451ff700 1 -- 172.21.15.57:0/1137968963 >> 172.21.15.57:0/1137968963 conn(0x7f7e400fbca0 msgr2=0x7f7e400fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:08.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.111+0000 7f7e451ff700 1 -- 172.21.15.57:0/1137968963 shutdown_connections 2023-07-20T18:58:08.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.111+0000 7f7e451ff700 1 -- 172.21.15.57:0/1137968963 wait complete. 2023-07-20T18:58:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.113+0000 7f7e451ff700 1 Processor -- start 2023-07-20T18:58:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.113+0000 7f7e451ff700 1 -- start start 2023-07-20T18:58:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.113+0000 7f7e451ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e40100770 0x7f7e40074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e451ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e40101960 0x7f7e40072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e451ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e40102300 0x7f7e400730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e451ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e40073840 con 0x7f7e40102300 2023-07-20T18:58:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e451ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e400739b0 con 0x7f7e40101960 2023-07-20T18:58:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e451ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e401a05b0 con 0x7f7e40100770 2023-07-20T18:58:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3f59e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e40102300 0x7f7e400730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3f59e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e40102300 0x7f7e400730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48698/0 (socket says 172.21.15.57:48698) 2023-07-20T18:58:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3f59e700 1 -- 172.21.15.57:0/3010623722 learned_addr learned my addr 172.21.15.57:0/3010623722 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3e59c700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e40101960 0x7f7e40072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3ed9d700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e40100770 0x7f7e40074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3f59e700 1 -- 172.21.15.57:0/3010623722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e40100770 msgr2=0x7f7e40074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3f59e700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e40100770 0x7f7e40074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3f59e700 1 -- 172.21.15.57:0/3010623722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e40101960 msgr2=0x7f7e40072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3f59e700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e40101960 0x7f7e40072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3f59e700 1 -- 172.21.15.57:0/3010623722 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e300095f0 con 0x7f7e40102300 2023-07-20T18:58:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.114+0000 7f7e3e59c700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e40101960 0x7f7e40072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.115+0000 7f7e3f59e700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e40102300 0x7f7e400730a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7e38007ee0 tx=0x7f7e3800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:08.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.115+0000 7f7e27fff700 1 -- 172.21.15.57:0/3010623722 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3800dd20 con 0x7f7e40102300 2023-07-20T18:58:08.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.115+0000 7f7e451ff700 1 -- 172.21.15.57:0/3010623722 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e401a0890 con 0x7f7e40102300 2023-07-20T18:58:08.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.115+0000 7f7e27fff700 1 -- 172.21.15.57:0/3010623722 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f7e3800f040 con 0x7f7e40102300 2023-07-20T18:58:08.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.115+0000 7f7e27fff700 1 -- 172.21.15.57:0/3010623722 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38015400 con 0x7f7e40102300 2023-07-20T18:58:08.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.115+0000 7f7e451ff700 1 -- 172.21.15.57:0/3010623722 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e401a0da0 con 0x7f7e40102300 2023-07-20T18:58:08.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.116+0000 7f7e451ff700 1 -- 172.21.15.57:0/3010623722 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4010c110 con 0x7f7e40102300 2023-07-20T18:58:08.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.118+0000 7f7e27fff700 1 -- 172.21.15.57:0/3010623722 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7e38003a00 con 0x7f7e40102300 2023-07-20T18:58:08.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.119+0000 7f7e27fff700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7e280a11a0 0x7f7e280a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:08.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.120+0000 7f7e27fff700 1 -- 172.21.15.57:0/3010623722 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7e380cbd60 con 0x7f7e40102300 2023-07-20T18:58:08.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.120+0000 7f7e3ed9d700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7e280a11a0 0x7f7e280a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:08.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.120+0000 7f7e3ed9d700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7e280a11a0 0x7f7e280a3660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000b580 tx=0x7f7e30005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:08.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.124+0000 7f7e27fff700 1 -- 172.21.15.57:0/3010623722 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7e3804e780 con 0x7f7e40102300 2023-07-20T18:58:08.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:07 smithi057 ceph-mon[126340]: Deploying daemon mds.e on smithi088 2023-07-20T18:58:08.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:07 smithi057 ceph-mon[126340]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:08.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:07 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.57:6836/3251478878,v1:172.21.15.57:6837/3251478878] up:boot 2023-07-20T18:58:08.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:07 smithi057 ceph-mon[126340]: fsmap 4 up:standby 2023-07-20T18:58:08.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:07 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T18:58:08.283 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 20 18:58:07 smithi088 podman[153467]: 2023-07-20T18:58:08.283 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 20 18:58:08 smithi088 ceph-mds[153508]: -- 172.21.15.88:0/308189415 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561b94c6b1e0 con 0x561b94c41000 2023-07-20T18:58:08.284 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 20 18:58:08 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-e[153483]: starting mds.e at 2023-07-20T18:58:08.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.381+0000 7f7e451ff700 1 -- 172.21.15.57:0/3010623722 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e40107600 con 0x7f7e40102300 2023-07-20T18:58:08.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.381+0000 7f7e27fff700 1 -- 172.21.15.57:0/3010623722 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3525 (secure 0 0 0) 0x7f7e380958b0 con 0x7f7e40102300 2023-07-20T18:58:08.431 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:08.432 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/3251478878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":3251478878},{"type":"v1","addr":"172.21.15.57:6837","nonce":3251478878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-20T18:58:08.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 -- 172.21.15.57:0/3010623722 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7e280a11a0 msgr2=0x7f7e280a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:08.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7e280a11a0 0x7f7e280a3660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000b580 tx=0x7f7e30005fb0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 -- 172.21.15.57:0/3010623722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e40102300 msgr2=0x7f7e400730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:08.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e40102300 0x7f7e400730a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7e38007ee0 tx=0x7f7e3800d3b0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 -- 172.21.15.57:0/3010623722 shutdown_connections 2023-07-20T18:58:08.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7e280a11a0 0x7f7e280a3660 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e40100770 0x7f7e40074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e40102300 0x7f7e400730a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 --2- 172.21.15.57:0/3010623722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e40101960 0x7f7e40072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:08.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 -- 172.21.15.57:0/3010623722 >> 172.21.15.57:0/3010623722 conn(0x7f7e400fbca0 msgr2=0x7f7e400fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:08.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 -- 172.21.15.57:0/3010623722 shutdown_connections 2023-07-20T18:58:08.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:08.384+0000 7f7e451ff700 1 -- 172.21.15.57:0/3010623722 wait complete. 2023-07-20T18:58:08.436 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 5 2023-07-20T18:58:08.631 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 20 18:58:08 smithi088 bash[153467]: 07e9cea9a6e2ffe584c8ed1cf204c259f7299ae502301be93614ecdc48bd9b26 2023-07-20T18:58:08.632 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 20 18:58:08 smithi088 systemd[1]: Started Ceph mds.e for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:09.632 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:09.632 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:09.632 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:09.632 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:09.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:09.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:09.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: Deploying daemon mds.f on smithi192 2023-07-20T18:58:09.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3010623722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:09.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:09.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.88:6834/1373743783,v1:172.21.15.88:6835/1373743783] up:boot 2023-07-20T18:58:09.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: fsmap 5 up:standby 2023-07-20T18:58:09.633 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:09 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T18:58:09.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:09.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:09.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:09.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:09.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:09.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:09.653 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: Deploying daemon mds.f on smithi192 2023-07-20T18:58:09.653 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3010623722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:09.653 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:09.653 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.88:6834/1373743783,v1:172.21.15.88:6835/1373743783] up:boot 2023-07-20T18:58:09.653 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: fsmap 5 up:standby 2023-07-20T18:58:09.653 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:09 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T18:58:09.734 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:09.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:09.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:09.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:09.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:09.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:09.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: Deploying daemon mds.f on smithi192 2023-07-20T18:58:09.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3010623722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:09.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:09.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.88:6834/1373743783,v1:172.21.15.88:6835/1373743783] up:boot 2023-07-20T18:58:09.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: fsmap 5 up:standby 2023-07-20T18:58:09.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:09 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T18:58:10.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Jul 20 18:58:09 smithi192 systemd[1]: Starting Ceph mds.f for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:10.453 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:10.467 INFO:journalctl@ceph.mds.f.smithi192.stdout:Jul 20 18:58:10 smithi192 podman[154732]: 2023-07-20T18:58:10.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.622+0000 7fcec21ff700 1 -- 172.21.15.57:0/2810742505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc102300 msgr2=0x7fcebc108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:10.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.622+0000 7fcec21ff700 1 --2- 172.21.15.57:0/2810742505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc102300 0x7fcebc108bb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009f90 tx=0x7fceb400b330 comp rx=0 tx=0).stop 2023-07-20T18:58:10.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.622+0000 7fcec21ff700 1 -- 172.21.15.57:0/2810742505 shutdown_connections 2023-07-20T18:58:10.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.622+0000 7fcec21ff700 1 --2- 172.21.15.57:0/2810742505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc102300 0x7fcebc108bb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.622+0000 7fcec21ff700 1 --2- 172.21.15.57:0/2810742505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcebc100770 0x7fcebc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.622+0000 7fcec21ff700 1 --2- 172.21.15.57:0/2810742505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcebc101960 0x7fcebc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.622+0000 7fcec21ff700 1 -- 172.21.15.57:0/2810742505 >> 172.21.15.57:0/2810742505 conn(0x7fcebc0fbca0 msgr2=0x7fcebc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.622+0000 7fcec21ff700 1 -- 172.21.15.57:0/2810742505 shutdown_connections 2023-07-20T18:58:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.622+0000 7fcec21ff700 1 -- 172.21.15.57:0/2810742505 wait complete. 2023-07-20T18:58:10.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.624+0000 7fcec21ff700 1 Processor -- start 2023-07-20T18:58:10.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.624+0000 7fcec21ff700 1 -- start start 2023-07-20T18:58:10.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.624+0000 7fcec21ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcebc100770 0x7fcebc1965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcec21ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc101960 0x7fcebc196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcec21ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcebc102300 0x7fcebc19db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcec21ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc19e150 con 0x7fcebc100770 2023-07-20T18:58:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcec21ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc19e290 con 0x7fcebc102300 2023-07-20T18:58:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcec21ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc19e420 con 0x7fcebc101960 2023-07-20T18:58:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebb7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcebc100770 0x7fcebc1965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebaffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc101960 0x7fcebc196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebaffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc101960 0x7fcebc196b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:38422/0 (socket says 172.21.15.57:38422) 2023-07-20T18:58:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebaffd700 1 -- 172.21.15.57:0/2309357760 learned_addr learned my addr 172.21.15.57:0/2309357760 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebb7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcebc100770 0x7fcebc1965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48720/0 (socket says 172.21.15.57:48720) 2023-07-20T18:58:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebbfff700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcebc102300 0x7fcebc19db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebaffd700 1 -- 172.21.15.57:0/2309357760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcebc102300 msgr2=0x7fcebc19db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebaffd700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcebc102300 0x7fcebc19db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebaffd700 1 -- 172.21.15.57:0/2309357760 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcebc100770 msgr2=0x7fcebc1965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebaffd700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcebc100770 0x7fcebc1965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebaffd700 1 -- 172.21.15.57:0/2309357760 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb4009c40 con 0x7fcebc101960 2023-07-20T18:58:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.625+0000 7fcebaffd700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc101960 0x7fcebc196b10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fceb000e9a0 tx=0x7fceb000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.626+0000 7fceb8ff9700 1 -- 172.21.15.57:0/2309357760 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0009790 con 0x7fcebc101960 2023-07-20T18:58:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.626+0000 7fceb8ff9700 1 -- 172.21.15.57:0/2309357760 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fceb000cbd0 con 0x7fcebc101960 2023-07-20T18:58:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.626+0000 7fcec21ff700 1 -- 172.21.15.57:0/2309357760 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc19e700 con 0x7fcebc101960 2023-07-20T18:58:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.626+0000 7fceb8ff9700 1 -- 172.21.15.57:0/2309357760 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb000eee0 con 0x7fcebc101960 2023-07-20T18:58:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.626+0000 7fcec21ff700 1 -- 172.21.15.57:0/2309357760 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc19ec10 con 0x7fcebc101960 2023-07-20T18:58:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.627+0000 7fcec21ff700 1 -- 172.21.15.57:0/2309357760 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc107be0 con 0x7fcebc101960 2023-07-20T18:58:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.629+0000 7fceb8ff9700 1 -- 172.21.15.57:0/2309357760 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fceb0010430 con 0x7fcebc101960 2023-07-20T18:58:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.630+0000 7fceb8ff9700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fcea40a11a0 0x7fcea40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.631+0000 7fceb8ff9700 1 -- 172.21.15.57:0/2309357760 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fceb0014070 con 0x7fcebc101960 2023-07-20T18:58:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.631+0000 7fcebb7fe700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fcea40a11a0 0x7fcea40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.631+0000 7fcebb7fe700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fcea40a11a0 0x7fcea40a3660 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcebc1017c0 tx=0x7fceac009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.632+0000 7fceb8ff9700 1 -- 172.21.15.57:0/2309357760 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fceb0096350 con 0x7fcebc101960 2023-07-20T18:58:10.839 INFO:journalctl@ceph.mds.f.smithi192.stdout:Jul 20 18:58:10 smithi192 bash[154732]: bc713ea339159038ab6f30ba6f62172adc6f3c873c09778caee6797cb6f7e788 2023-07-20T18:58:10.839 INFO:journalctl@ceph.mds.f.smithi192.stdout:Jul 20 18:58:10 smithi192 ceph-mds[154772]: -- 172.21.15.192:0/877837293 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a180f871e0 con 0x55a180f5cc00 2023-07-20T18:58:10.840 INFO:journalctl@ceph.mds.f.smithi192.stdout:Jul 20 18:58:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-f[154748]: starting mds.f at 2023-07-20T18:58:10.840 INFO:journalctl@ceph.mds.f.smithi192.stdout:Jul 20 18:58:10 smithi192 systemd[1]: Started Ceph mds.f for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.886+0000 7fcec21ff700 1 -- 172.21.15.57:0/2309357760 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcebc02ce00 con 0x7fcebc101960 2023-07-20T18:58:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.887+0000 7fceb8ff9700 1 -- 172.21.15.57:0/2309357760 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5042 (secure 0 0 0) 0x7fceb0099970 con 0x7fcebc101960 2023-07-20T18:58:10.937 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:10.937 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/3251478878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":3251478878},{"type":"v1","addr":"172.21.15.57:6837","nonce":3251478878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2219652585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2219652585},{"type":"v1","addr":"172.21.15.192:6835","nonce":2219652585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1373743783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1373743783},{"type":"v1","addr":"172.21.15.88:6835","nonce":1373743783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-20T18:58:10.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.889+0000 7fcec21ff700 1 -- 172.21.15.57:0/2309357760 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fcea40a11a0 msgr2=0x7fcea40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:10.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.889+0000 7fcec21ff700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fcea40a11a0 0x7fcea40a3660 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcebc1017c0 tx=0x7fceac009310 comp rx=0 tx=0).stop 2023-07-20T18:58:10.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.889+0000 7fcec21ff700 1 -- 172.21.15.57:0/2309357760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc101960 msgr2=0x7fcebc196b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:10.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.889+0000 7fcec21ff700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc101960 0x7fcebc196b10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fceb000e9a0 tx=0x7fceb000ed60 comp rx=0 tx=0).stop 2023-07-20T18:58:10.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.890+0000 7fcec21ff700 1 -- 172.21.15.57:0/2309357760 shutdown_connections 2023-07-20T18:58:10.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.890+0000 7fcec21ff700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fcea40a11a0 0x7fcea40a3660 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:10.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.890+0000 7fcec21ff700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcebc102300 0x7fcebc19db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:10.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.890+0000 7fcec21ff700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcebc100770 0x7fcebc1965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:10.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.890+0000 7fcec21ff700 1 --2- 172.21.15.57:0/2309357760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc101960 0x7fcebc196b10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:10.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.890+0000 7fcec21ff700 1 -- 172.21.15.57:0/2309357760 >> 172.21.15.57:0/2309357760 conn(0x7fcebc0fbca0 msgr2=0x7fcebc104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:10.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.890+0000 7fcec21ff700 1 -- 172.21.15.57:0/2309357760 shutdown_connections 2023-07-20T18:58:10.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:10.890+0000 7fcec21ff700 1 -- 172.21.15.57:0/2309357760 wait complete. 2023-07-20T18:58:10.941 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 7 2023-07-20T18:58:11.767 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:11.768 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:11.768 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:11.768 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:11.768 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:11.768 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:11.768 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: Deploying daemon mds.c on smithi192 2023-07-20T18:58:11.769 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.192:6834/2219652585,v1:172.21.15.192:6835/2219652585] up:boot 2023-07-20T18:58:11.769 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: fsmap 6 up:standby 2023-07-20T18:58:11.769 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T18:58:11.769 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:11.769 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:11 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2309357760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:11.882 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:11.882 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:11.882 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:11.882 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:11.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:11.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:11.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: Deploying daemon mds.c on smithi192 2023-07-20T18:58:11.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.192:6834/2219652585,v1:172.21.15.192:6835/2219652585] up:boot 2023-07-20T18:58:11.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: fsmap 6 up:standby 2023-07-20T18:58:11.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T18:58:11.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:11.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:11 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2309357760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:11.984 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:11.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:11.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:11.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:11.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:11.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:11.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: Deploying daemon mds.c on smithi192 2023-07-20T18:58:11.986 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.192:6834/2219652585,v1:172.21.15.192:6835/2219652585] up:boot 2023-07-20T18:58:11.986 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: fsmap 6 up:standby 2023-07-20T18:58:11.986 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T18:58:11.986 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:11.986 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:11 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2309357760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:12.089 INFO:journalctl@ceph.mds.c.smithi192.stdout:Jul 20 18:58:11 smithi192 systemd[1]: Starting Ceph mds.c for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:12.572 INFO:journalctl@ceph.mds.c.smithi192.stdout:Jul 20 18:58:12 smithi192 podman[155072]: 2023-07-20T18:58:12.572 INFO:journalctl@ceph.mds.c.smithi192.stdout:Jul 20 18:58:12 smithi192 ceph-mds[155112]: -- 172.21.15.192:0/3366306432 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a4fa0dd1e0 con 0x55a4fa0b3000 2023-07-20T18:58:12.839 INFO:journalctl@ceph.mds.c.smithi192.stdout:Jul 20 18:58:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-c[155088]: starting mds.c at 2023-07-20T18:58:12.839 INFO:journalctl@ceph.mds.c.smithi192.stdout:Jul 20 18:58:12 smithi192 bash[155072]: 492b8243bfa2522c3e01def020fe5f403616c43987b756521eda7033013cf500 2023-07-20T18:58:12.839 INFO:journalctl@ceph.mds.c.smithi192.stdout:Jul 20 18:58:12 smithi192 systemd[1]: Started Ceph mds.c for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:12.959 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.124+0000 7f9d23dff700 1 -- 172.21.15.57:0/199758114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d1c100770 msgr2=0x7f9d1c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.124+0000 7f9d23dff700 1 --2- 172.21.15.57:0/199758114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d1c100770 0x7f9d1c100b90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9d180099c0 tx=0x7f9d18009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.124+0000 7f9d23dff700 1 -- 172.21.15.57:0/199758114 shutdown_connections 2023-07-20T18:58:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.124+0000 7f9d23dff700 1 --2- 172.21.15.57:0/199758114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d1c102300 0x7f9d1c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.124+0000 7f9d23dff700 1 --2- 172.21.15.57:0/199758114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d1c100770 0x7f9d1c100b90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.124+0000 7f9d23dff700 1 --2- 172.21.15.57:0/199758114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d1c101960 0x7f9d1c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.124+0000 7f9d23dff700 1 -- 172.21.15.57:0/199758114 >> 172.21.15.57:0/199758114 conn(0x7f9d1c0fbca0 msgr2=0x7f9d1c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.124+0000 7f9d23dff700 1 -- 172.21.15.57:0/199758114 shutdown_connections 2023-07-20T18:58:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.124+0000 7f9d23dff700 1 -- 172.21.15.57:0/199758114 wait complete. 2023-07-20T18:58:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.125+0000 7f9d23dff700 1 Processor -- start 2023-07-20T18:58:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.125+0000 7f9d23dff700 1 -- start start 2023-07-20T18:58:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d23dff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d1c100770 0x7f9d1c074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d23dff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d1c101960 0x7f9d1c072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d23dff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d1c102300 0x7f9d1c0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d23dff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c073840 con 0x7f9d1c102300 2023-07-20T18:58:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d23dff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c0739b0 con 0x7f9d1c100770 2023-07-20T18:58:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d23dff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c1a0500 con 0x7f9d1c101960 2023-07-20T18:58:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d2239c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d1c102300 0x7f9d1c0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d2239c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d1c102300 0x7f9d1c0730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48732/0 (socket says 172.21.15.57:48732) 2023-07-20T18:58:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d2239c700 1 -- 172.21.15.57:0/469810990 learned_addr learned my addr 172.21.15.57:0/469810990 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d2139a700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d1c101960 0x7f9d1c072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d21b9b700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d1c100770 0x7f9d1c074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d2239c700 1 -- 172.21.15.57:0/469810990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d1c101960 msgr2=0x7f9d1c072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d2239c700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d1c101960 0x7f9d1c072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d2239c700 1 -- 172.21.15.57:0/469810990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d1c100770 msgr2=0x7f9d1c074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d2239c700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d1c100770 0x7f9d1c074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.126+0000 7f9d2239c700 1 -- 172.21.15.57:0/469810990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d180095f0 con 0x7f9d1c102300 2023-07-20T18:58:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.127+0000 7f9d2239c700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d1c102300 0x7f9d1c0730a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400ef30 tx=0x7f9d1400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.127+0000 7f9d12ffd700 1 -- 172.21.15.57:0/469810990 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14018070 con 0x7f9d1c102300 2023-07-20T18:58:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.127+0000 7f9d23dff700 1 -- 172.21.15.57:0/469810990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1c1a0780 con 0x7f9d1c102300 2023-07-20T18:58:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.127+0000 7f9d12ffd700 1 -- 172.21.15.57:0/469810990 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f9d1400f040 con 0x7f9d1c102300 2023-07-20T18:58:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.127+0000 7f9d12ffd700 1 -- 172.21.15.57:0/469810990 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1400cd50 con 0x7f9d1c102300 2023-07-20T18:58:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.127+0000 7f9d23dff700 1 -- 172.21.15.57:0/469810990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1c1a0c40 con 0x7f9d1c102300 2023-07-20T18:58:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.128+0000 7f9d23dff700 1 -- 172.21.15.57:0/469810990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1c10c0c0 con 0x7f9d1c102300 2023-07-20T18:58:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.130+0000 7f9d12ffd700 1 -- 172.21.15.57:0/469810990 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9d14003a00 con 0x7f9d1c102300 2023-07-20T18:58:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.132+0000 7f9d12ffd700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9d080a11a0 0x7f9d080a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.132+0000 7f9d12ffd700 1 -- 172.21.15.57:0/469810990 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9d140cca60 con 0x7f9d1c102300 2023-07-20T18:58:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.132+0000 7f9d21b9b700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9d080a11a0 0x7f9d080a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:13.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.133+0000 7f9d21b9b700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9d080a11a0 0x7f9d080a3660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800b580 tx=0x7f9d18005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:13.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.134+0000 7f9d12ffd700 1 -- 172.21.15.57:0/469810990 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9d14096640 con 0x7f9d1c102300 2023-07-20T18:58:13.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.388+0000 7f9d23dff700 1 -- 172.21.15.57:0/469810990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d1c1075f0 con 0x7f9d1c102300 2023-07-20T18:58:13.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.389+0000 7f9d12ffd700 1 -- 172.21.15.57:0/469810990 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5802 (secure 0 0 0) 0x7f9d14099c60 con 0x7f9d1c102300 2023-07-20T18:58:13.439 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:13.439 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/3251478878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":3251478878},{"type":"v1","addr":"172.21.15.57:6837","nonce":3251478878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2219652585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2219652585},{"type":"v1","addr":"172.21.15.192:6835","nonce":2219652585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1373743783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1373743783},{"type":"v1","addr":"172.21.15.88:6835","nonce":1373743783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2391717255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2391717255},{"type":"v1","addr":"172.21.15.192:6837","nonce":2391717255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T18:58:13.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 -- 172.21.15.57:0/469810990 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9d080a11a0 msgr2=0x7f9d080a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:13.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9d080a11a0 0x7f9d080a3660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800b580 tx=0x7f9d18005fb0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 -- 172.21.15.57:0/469810990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d1c102300 msgr2=0x7f9d1c0730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:13.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d1c102300 0x7f9d1c0730a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400ef30 tx=0x7f9d1400c3a0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 -- 172.21.15.57:0/469810990 shutdown_connections 2023-07-20T18:58:13.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9d080a11a0 0x7f9d080a3660 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d1c102300 0x7f9d1c0730a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d1c100770 0x7f9d1c074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 --2- 172.21.15.57:0/469810990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d1c101960 0x7f9d1c072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:13.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 -- 172.21.15.57:0/469810990 >> 172.21.15.57:0/469810990 conn(0x7f9d1c0fbca0 msgr2=0x7f9d1c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:13.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.391+0000 7f9d23dff700 1 -- 172.21.15.57:0/469810990 shutdown_connections 2023-07-20T18:58:13.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:13.392+0000 7f9d23dff700 1 -- 172.21.15.57:0/469810990 wait complete. 2023-07-20T18:58:13.444 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 8 2023-07-20T18:58:13.734 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.192:6836/2391717255,v1:172.21.15.192:6837/2391717255] up:boot 2023-07-20T18:58:13.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: fsmap 7 up:standby 2023-07-20T18:58:13.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T18:58:13.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:13.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:13.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:13.735 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:13.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:13.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:13.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: Deploying daemon mds.h on smithi088 2023-07-20T18:58:13.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:13.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:13 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/469810990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:13.882 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.192:6836/2391717255,v1:172.21.15.192:6837/2391717255] up:boot 2023-07-20T18:58:13.882 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: fsmap 7 up:standby 2023-07-20T18:58:13.882 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T18:58:13.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:13.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:13.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:13.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:13.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:13.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:13.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: Deploying daemon mds.h on smithi088 2023-07-20T18:58:13.884 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:13.884 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:13 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/469810990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:14.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.192:6836/2391717255,v1:172.21.15.192:6837/2391717255] up:boot 2023-07-20T18:58:14.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: fsmap 7 up:standby 2023-07-20T18:58:14.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T18:58:14.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:14.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:14.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:14.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:14.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:14.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:14.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: Deploying daemon mds.h on smithi088 2023-07-20T18:58:14.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:14.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:13 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/469810990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:14.382 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 20 18:58:14 smithi088 systemd[1]: Starting Ceph mds.h for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:14.715 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 20 18:58:14 smithi088 podman[153814]: 2023-07-20T18:58:14.715 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 20 18:58:14 smithi088 ceph-mds[153855]: -- 172.21.15.88:0/943694753 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5558564cf1e0 con 0x5558564a5000 2023-07-20T18:58:14.716 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 20 18:58:14 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-h[153830]: starting mds.h at 2023-07-20T18:58:15.132 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 20 18:58:14 smithi088 bash[153814]: 1f5af1f3b8feecaa8e702061ad71baadae81caa147d54daa2238f09001d772fd 2023-07-20T18:58:15.132 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 20 18:58:14 smithi088 systemd[1]: Started Ceph mds.h for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:15.461 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:15.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.626+0000 7fd28e9ff700 1 -- 172.21.15.57:0/3930882221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd288101960 msgr2=0x7fd288101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:15.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.626+0000 7fd28e9ff700 1 --2- 172.21.15.57:0/3930882221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd288101960 0x7fd288101dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd2700099c0 tx=0x7fd270009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:15.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.626+0000 7fd28e9ff700 1 -- 172.21.15.57:0/3930882221 shutdown_connections 2023-07-20T18:58:15.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.626+0000 7fd28e9ff700 1 --2- 172.21.15.57:0/3930882221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288100770 0x7fd288100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:15.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.626+0000 7fd28e9ff700 1 --2- 172.21.15.57:0/3930882221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd288102300 0x7fd288108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:15.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.626+0000 7fd28e9ff700 1 --2- 172.21.15.57:0/3930882221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd288101960 0x7fd288101dc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:15.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.626+0000 7fd28e9ff700 1 -- 172.21.15.57:0/3930882221 >> 172.21.15.57:0/3930882221 conn(0x7fd2880fbca0 msgr2=0x7fd2880fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:15.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.626+0000 7fd28e9ff700 1 -- 172.21.15.57:0/3930882221 shutdown_connections 2023-07-20T18:58:15.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.626+0000 7fd28e9ff700 1 -- 172.21.15.57:0/3930882221 wait complete. 2023-07-20T18:58:15.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.627+0000 7fd28e9ff700 1 Processor -- start 2023-07-20T18:58:15.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd28e9ff700 1 -- start start 2023-07-20T18:58:15.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd28e9ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd288100770 0x7fd288196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:15.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd28e9ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288101960 0x7fd288196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:15.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd28e9ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd288102300 0x7fd28819dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd28e9ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28819e080 con 0x7fd288102300 2023-07-20T18:58:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd28e9ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28819e1c0 con 0x7fd288100770 2023-07-20T18:58:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd28e9ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28819e350 con 0x7fd288101960 2023-07-20T18:58:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd2877fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288101960 0x7fd288196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd287fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd288100770 0x7fd288196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd2877fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288101960 0x7fd288196a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:36712/0 (socket says 172.21.15.57:36712) 2023-07-20T18:58:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.628+0000 7fd2877fe700 1 -- 172.21.15.57:0/90557120 learned_addr learned my addr 172.21.15.57:0/90557120 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:15.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.629+0000 7fd28cf9c700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd288102300 0x7fd28819dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.629+0000 7fd2877fe700 1 -- 172.21.15.57:0/90557120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd288100770 msgr2=0x7fd288196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.629+0000 7fd2877fe700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd288100770 0x7fd288196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.629+0000 7fd2877fe700 1 -- 172.21.15.57:0/90557120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd288102300 msgr2=0x7fd28819dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.629+0000 7fd2877fe700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd288102300 0x7fd28819dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.629+0000 7fd2877fe700 1 -- 172.21.15.57:0/90557120 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2700095f0 con 0x7fd288101960 2023-07-20T18:58:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.629+0000 7fd2877fe700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288101960 0x7fd288196a40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd27000c010 tx=0x7fd27000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.630+0000 7fd2857fa700 1 -- 172.21.15.57:0/90557120 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27001d070 con 0x7fd288101960 2023-07-20T18:58:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.630+0000 7fd2857fa700 1 -- 172.21.15.57:0/90557120 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fd270005050 con 0x7fd288101960 2023-07-20T18:58:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.630+0000 7fd2857fa700 1 -- 172.21.15.57:0/90557120 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270004bd0 con 0x7fd288101960 2023-07-20T18:58:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.630+0000 7fd28e9ff700 1 -- 172.21.15.57:0/90557120 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28819e5d0 con 0x7fd288101960 2023-07-20T18:58:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.630+0000 7fd28e9ff700 1 -- 172.21.15.57:0/90557120 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28819e9f0 con 0x7fd288101960 2023-07-20T18:58:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.631+0000 7fd28e9ff700 1 -- 172.21.15.57:0/90557120 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd288107c90 con 0x7fd288101960 2023-07-20T18:58:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.633+0000 7fd2857fa700 1 -- 172.21.15.57:0/90557120 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd270004d30 con 0x7fd288101960 2023-07-20T18:58:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.634+0000 7fd2857fa700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd2740a1220 0x7fd2740a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.634+0000 7fd2857fa700 1 -- 172.21.15.57:0/90557120 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd2700cd320 con 0x7fd288101960 2023-07-20T18:58:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.635+0000 7fd287fff700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd2740a1220 0x7fd2740a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.635+0000 7fd287fff700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd2740a1220 0x7fd2740a36e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2881017c0 tx=0x7fd278009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:15.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.636+0000 7fd2857fa700 1 -- 172.21.15.57:0/90557120 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd270096f00 con 0x7fd288101960 2023-07-20T18:58:15.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.891+0000 7fd28e9ff700 1 -- 172.21.15.57:0/90557120 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd28802c970 con 0x7fd288101960 2023-07-20T18:58:15.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.893+0000 7fd2857fa700 1 -- 172.21.15.57:0/90557120 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6556 (secure 0 0 0) 0x7fd27009a520 con 0x7fd288101960 2023-07-20T18:58:15.942 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:15.942 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/3251478878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":3251478878},{"type":"v1","addr":"172.21.15.57:6837","nonce":3251478878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2219652585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2219652585},{"type":"v1","addr":"172.21.15.192:6835","nonce":2219652585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1373743783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1373743783},{"type":"v1","addr":"172.21.15.88:6835","nonce":1373743783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2391717255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2391717255},{"type":"v1","addr":"172.21.15.192:6837","nonce":2391717255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/678288704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":678288704},{"type":"v1","addr":"172.21.15.88:6837","nonce":678288704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-20T18:58:15.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 -- 172.21.15.57:0/90557120 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd2740a1220 msgr2=0x7fd2740a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:15.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd2740a1220 0x7fd2740a36e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2881017c0 tx=0x7fd278009260 comp rx=0 tx=0).stop 2023-07-20T18:58:15.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 -- 172.21.15.57:0/90557120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288101960 msgr2=0x7fd288196a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:15.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288101960 0x7fd288196a40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd27000c010 tx=0x7fd27000f690 comp rx=0 tx=0).stop 2023-07-20T18:58:15.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 -- 172.21.15.57:0/90557120 shutdown_connections 2023-07-20T18:58:15.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd2740a1220 0x7fd2740a36e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:15.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd288102300 0x7fd28819dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:15.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd288100770 0x7fd288196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:15.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 --2- 172.21.15.57:0/90557120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288101960 0x7fd288196a40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:15.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 -- 172.21.15.57:0/90557120 >> 172.21.15.57:0/90557120 conn(0x7fd2880fbca0 msgr2=0x7fd288104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:15.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.894+0000 7fd28e9ff700 1 -- 172.21.15.57:0/90557120 shutdown_connections 2023-07-20T18:58:15.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:15.895+0000 7fd28e9ff700 1 -- 172.21.15.57:0/90557120 wait complete. 2023-07-20T18:58:15.947 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2023-07-20T18:58:15.960 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:15.961 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.88:6836/678288704,v1:172.21.15.88:6837/678288704] up:boot 2023-07-20T18:58:15.961 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: fsmap 8 up:standby 2023-07-20T18:58:15.961 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T18:58:15.961 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:15.961 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:15.961 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:15.962 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:15.962 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:15.962 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:15.962 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:15 smithi057 ceph-mon[126340]: Deploying daemon mds.i on smithi192 2023-07-20T18:58:16.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:16.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.88:6836/678288704,v1:172.21.15.88:6837/678288704] up:boot 2023-07-20T18:58:16.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: fsmap 8 up:standby 2023-07-20T18:58:16.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T18:58:16.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:16.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:16.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:16.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:16.092 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:16.092 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:16.092 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:15 smithi192 ceph-mon[129200]: Deploying daemon mds.i on smithi192 2023-07-20T18:58:16.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:16.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.88:6836/678288704,v1:172.21.15.88:6837/678288704] up:boot 2023-07-20T18:58:16.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: fsmap 8 up:standby 2023-07-20T18:58:16.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T18:58:16.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:16.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:16.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:16.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:16.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:16.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:16.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:15 smithi088 ceph-mon[128823]: Deploying daemon mds.i on smithi192 2023-07-20T18:58:16.590 INFO:journalctl@ceph.mds.i.smithi192.stdout:Jul 20 18:58:16 smithi192 systemd[1]: Starting Ceph mds.i for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:16.915 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:16 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/90557120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:16.916 INFO:journalctl@ceph.mds.i.smithi192.stdout:Jul 20 18:58:16 smithi192 podman[155412]: 2023-07-20T18:58:16.916 INFO:journalctl@ceph.mds.i.smithi192.stdout:Jul 20 18:58:16 smithi192 ceph-mds[155452]: -- 172.21.15.192:0/3042489513 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562ee19eb1e0 con 0x562ee19c0800 2023-07-20T18:58:16.916 INFO:journalctl@ceph.mds.i.smithi192.stdout:Jul 20 18:58:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-i[155428]: starting mds.i at 2023-07-20T18:58:16.916 INFO:journalctl@ceph.mds.i.smithi192.stdout:Jul 20 18:58:16 smithi192 bash[155412]: 37213b8809048c71cf31d212f9ae7daa7024e82441c5b81b4143d8080393ca2c 2023-07-20T18:58:17.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:16 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/90557120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:17.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:16 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/90557120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:17.339 INFO:journalctl@ceph.mds.i.smithi192.stdout:Jul 20 18:58:16 smithi192 systemd[1]: Started Ceph mds.i for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:17.964 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:18.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.131+0000 7fe97adff700 1 -- 172.21.15.57:0/245465257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe974101960 msgr2=0x7fe974101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:18.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.131+0000 7fe97adff700 1 --2- 172.21.15.57:0/245465257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe974101960 0x7fe974101dc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe9600099c0 tx=0x7fe960009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.132+0000 7fe97adff700 1 -- 172.21.15.57:0/245465257 shutdown_connections 2023-07-20T18:58:18.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.132+0000 7fe97adff700 1 --2- 172.21.15.57:0/245465257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe974102300 0x7fe974108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.132+0000 7fe97adff700 1 --2- 172.21.15.57:0/245465257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe974100770 0x7fe974100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.132+0000 7fe97adff700 1 --2- 172.21.15.57:0/245465257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe974101960 0x7fe974101dc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.132+0000 7fe97adff700 1 -- 172.21.15.57:0/245465257 >> 172.21.15.57:0/245465257 conn(0x7fe9740fbca0 msgr2=0x7fe9740fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:18.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.132+0000 7fe97adff700 1 -- 172.21.15.57:0/245465257 shutdown_connections 2023-07-20T18:58:18.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.132+0000 7fe97adff700 1 -- 172.21.15.57:0/245465257 wait complete. 2023-07-20T18:58:18.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.133+0000 7fe97adff700 1 Processor -- start 2023-07-20T18:58:18.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.133+0000 7fe97adff700 1 -- start start 2023-07-20T18:58:18.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.133+0000 7fe97adff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe974100770 0x7fe974198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.133+0000 7fe97adff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe974101960 0x7fe974198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.133+0000 7fe97adff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe974102300 0x7fe97419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.133+0000 7fe97adff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9741a02b0 con 0x7fe974101960 2023-07-20T18:58:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.133+0000 7fe97adff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9741a03f0 con 0x7fe974100770 2023-07-20T18:58:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.133+0000 7fe97adff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9741a0580 con 0x7fe974102300 2023-07-20T18:58:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.134+0000 7fe978b9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe974100770 0x7fe974198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.134+0000 7fe978b9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe974100770 0x7fe974198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:47334/0 (socket says 172.21.15.57:47334) 2023-07-20T18:58:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.134+0000 7fe978b9b700 1 -- 172.21.15.57:0/2248669680 learned_addr learned my addr 172.21.15.57:0/2248669680 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:18.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.134+0000 7fe973fff700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe974101960 0x7fe974198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:18.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.134+0000 7fe97939c700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe974102300 0x7fe97419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:18.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.134+0000 7fe978b9b700 1 -- 172.21.15.57:0/2248669680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe974102300 msgr2=0x7fe97419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:18.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.134+0000 7fe978b9b700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe974102300 0x7fe97419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.134+0000 7fe978b9b700 1 -- 172.21.15.57:0/2248669680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe974101960 msgr2=0x7fe974198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:18.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.134+0000 7fe978b9b700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe974101960 0x7fe974198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.134+0000 7fe978b9b700 1 -- 172.21.15.57:0/2248669680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9600095f0 con 0x7fe974100770 2023-07-20T18:58:18.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.135+0000 7fe978b9b700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe974100770 0x7fe974198730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe96800e970 tx=0x7fe96800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:18.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.135+0000 7fe971ffb700 1 -- 172.21.15.57:0/2248669680 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe968009790 con 0x7fe974100770 2023-07-20T18:58:18.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.135+0000 7fe971ffb700 1 -- 172.21.15.57:0/2248669680 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fe96800eec0 con 0x7fe974100770 2023-07-20T18:58:18.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.135+0000 7fe97adff700 1 -- 172.21.15.57:0/2248669680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9741a0860 con 0x7fe974100770 2023-07-20T18:58:18.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.135+0000 7fe971ffb700 1 -- 172.21.15.57:0/2248669680 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe968003d90 con 0x7fe974100770 2023-07-20T18:58:18.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.135+0000 7fe97adff700 1 -- 172.21.15.57:0/2248669680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9741a0d70 con 0x7fe974100770 2023-07-20T18:58:18.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.136+0000 7fe97adff700 1 -- 172.21.15.57:0/2248669680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe97410c110 con 0x7fe974100770 2023-07-20T18:58:18.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.138+0000 7fe971ffb700 1 -- 172.21.15.57:0/2248669680 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe968004000 con 0x7fe974100770 2023-07-20T18:58:18.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.139+0000 7fe971ffb700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe9640a11a0 0x7fe9640a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:18.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.139+0000 7fe971ffb700 1 -- 172.21.15.57:0/2248669680 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe968014070 con 0x7fe974100770 2023-07-20T18:58:18.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.140+0000 7fe973fff700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe9640a11a0 0x7fe9640a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:18.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.140+0000 7fe973fff700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe9640a11a0 0x7fe9640a3660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe960000c00 tx=0x7fe960005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:18.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.143+0000 7fe971ffb700 1 -- 172.21.15.57:0/2248669680 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe9680962f0 con 0x7fe974100770 2023-07-20T18:58:18.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:18.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:18.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:18.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:18.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:18.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:18.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:18.237 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: Deploying daemon mds.g on smithi057 2023-07-20T18:58:18.237 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.192:6838/1456232015,v1:172.21.15.192:6839/1456232015] up:boot 2023-07-20T18:58:18.237 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: fsmap 9 up:standby 2023-07-20T18:58:18.237 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:17 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T18:58:18.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:18.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:18.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:18.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:18.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:18.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:18.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:18.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: Deploying daemon mds.g on smithi057 2023-07-20T18:58:18.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.192:6838/1456232015,v1:172.21.15.192:6839/1456232015] up:boot 2023-07-20T18:58:18.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: fsmap 9 up:standby 2023-07-20T18:58:18.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:17 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T18:58:18.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:18.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:18.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:18.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:18.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:18.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:18.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:18.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: Deploying daemon mds.g on smithi057 2023-07-20T18:58:18.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.192:6838/1456232015,v1:172.21.15.192:6839/1456232015] up:boot 2023-07-20T18:58:18.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: fsmap 9 up:standby 2023-07-20T18:58:18.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:17 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T18:58:18.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.406+0000 7fe97adff700 1 -- 172.21.15.57:0/2248669680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe97410c2d0 con 0x7fe974100770 2023-07-20T18:58:18.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.407+0000 7fe971ffb700 1 -- 172.21.15.57:0/2248669680 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7318 (secure 0 0 0) 0x7fe968099910 con 0x7fe974100770 2023-07-20T18:58:18.456 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:18.457 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/3251478878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":3251478878},{"type":"v1","addr":"172.21.15.57:6837","nonce":3251478878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2219652585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2219652585},{"type":"v1","addr":"172.21.15.192:6835","nonce":2219652585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1373743783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1373743783},{"type":"v1","addr":"172.21.15.88:6835","nonce":1373743783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2391717255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2391717255},{"type":"v1","addr":"172.21.15.192:6837","nonce":2391717255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/678288704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":678288704},{"type":"v1","addr":"172.21.15.88:6837","nonce":678288704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/1456232015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1456232015},{"type":"v1","addr":"172.21.15.192:6839","nonce":1456232015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T18:58:18.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 -- 172.21.15.57:0/2248669680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe9640a11a0 msgr2=0x7fe9640a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:18.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe9640a11a0 0x7fe9640a3660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe960000c00 tx=0x7fe960005fb0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 -- 172.21.15.57:0/2248669680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe974100770 msgr2=0x7fe974198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:18.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe974100770 0x7fe974198730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe96800e970 tx=0x7fe96800ec80 comp rx=0 tx=0).stop 2023-07-20T18:58:18.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 -- 172.21.15.57:0/2248669680 shutdown_connections 2023-07-20T18:58:18.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe9640a11a0 0x7fe9640a3660 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe974102300 0x7fe97419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe974100770 0x7fe974198730 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 --2- 172.21.15.57:0/2248669680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe974101960 0x7fe974198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:18.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 -- 172.21.15.57:0/2248669680 >> 172.21.15.57:0/2248669680 conn(0x7fe9740fbca0 msgr2=0x7fe974107190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:18.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 -- 172.21.15.57:0/2248669680 shutdown_connections 2023-07-20T18:58:18.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:18.410+0000 7fe97adff700 1 -- 172.21.15.57:0/2248669680 wait complete. 2023-07-20T18:58:18.461 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 10 2023-07-20T18:58:19.234 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:18 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2248669680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:19.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:18 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2248669680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:19.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:18 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2248669680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:19.485 INFO:journalctl@ceph.mds.g.smithi057.stdout:Jul 20 18:58:19 smithi057 systemd[1]: Starting Ceph mds.g for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:20.003 INFO:journalctl@ceph.mds.g.smithi057.stdout:Jul 20 18:58:19 smithi057 podman[160770]: 2023-07-20T18:58:20.003 INFO:journalctl@ceph.mds.g.smithi057.stdout:Jul 20 18:58:19 smithi057 ceph-mds[160810]: -- 172.21.15.57:0/2223460694 <== mon.0 v2:172.21.15.57:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eeabcb91e0 con 0x55eeabc8f000 2023-07-20T18:58:20.004 INFO:journalctl@ceph.mds.g.smithi057.stdout:Jul 20 18:58:19 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-g[160786]: starting mds.g at 2023-07-20T18:58:20.004 INFO:journalctl@ceph.mds.g.smithi057.stdout:Jul 20 18:58:19 smithi057 bash[160770]: 23b6240c82a79fc46143103a30a5a7ef05409e7459a65795c0a0de399d4ffd1b 2023-07-20T18:58:20.005 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:19 smithi057 ceph-mon[126340]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:20.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:19 smithi192 ceph-mon[129200]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:20.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:19 smithi088 ceph-mon[128823]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:20.478 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:20.484 INFO:journalctl@ceph.mds.g.smithi057.stdout:Jul 20 18:58:19 smithi057 systemd[1]: Started Ceph mds.g for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.665+0000 7f58e55ff700 1 -- 172.21.15.57:0/2625618309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58e0101270 msgr2=0x7f58e0103660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.665+0000 7f58e55ff700 1 --2- 172.21.15.57:0/2625618309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58e0101270 0x7f58e0103660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009990 tx=0x7f58d0009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.665+0000 7f58e55ff700 1 -- 172.21.15.57:0/2625618309 shutdown_connections 2023-07-20T18:58:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.665+0000 7f58e55ff700 1 --2- 172.21.15.57:0/2625618309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58e0101270 0x7f58e0103660 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.665+0000 7f58e55ff700 1 --2- 172.21.15.57:0/2625618309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e01064d0 0x7f58e01088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.665+0000 7f58e55ff700 1 --2- 172.21.15.57:0/2625618309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0103ba0 0x7f58e0105f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:20.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.665+0000 7f58e55ff700 1 -- 172.21.15.57:0/2625618309 >> 172.21.15.57:0/2625618309 conn(0x7f58e00faba0 msgr2=0x7f58e00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:20.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.665+0000 7f58e55ff700 1 -- 172.21.15.57:0/2625618309 shutdown_connections 2023-07-20T18:58:20.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.665+0000 7f58e55ff700 1 -- 172.21.15.57:0/2625618309 wait complete. 2023-07-20T18:58:20.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58e55ff700 1 Processor -- start 2023-07-20T18:58:20.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58e55ff700 1 -- start start 2023-07-20T18:58:20.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58e55ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0101270 0x7f58e006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:20.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58e55ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58e0103ba0 0x7f58e006f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:20.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58e55ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e01064d0 0x7f58e01a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:20.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58e55ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e00725d0 con 0x7f58e01064d0 2023-07-20T18:58:20.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58e55ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e0072740 con 0x7f58e0101270 2023-07-20T18:58:20.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58e55ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e01a6c90 con 0x7f58e0103ba0 2023-07-20T18:58:20.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58d7fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58e0103ba0 0x7f58e006f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:20.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58d7fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58e0103ba0 0x7f58e006f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:36788/0 (socket says 172.21.15.57:36788) 2023-07-20T18:58:20.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58d7fff700 1 -- 172.21.15.57:0/1411044751 learned_addr learned my addr 172.21.15.57:0/1411044751 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:20.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58deffd700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0101270 0x7f58e006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:20.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.667+0000 7f58df7fe700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e01064d0 0x7f58e01a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:20.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.668+0000 7f58d7fff700 1 -- 172.21.15.57:0/1411044751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0101270 msgr2=0x7f58e006efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:20.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.668+0000 7f58d7fff700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0101270 0x7f58e006efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:20.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.668+0000 7f58d7fff700 1 -- 172.21.15.57:0/1411044751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e01064d0 msgr2=0x7f58e01a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:20.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.668+0000 7f58d7fff700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e01064d0 0x7f58e01a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:20.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.668+0000 7f58d7fff700 1 -- 172.21.15.57:0/1411044751 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d8009c40 con 0x7f58e0103ba0 2023-07-20T18:58:20.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.668+0000 7f58d7fff700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58e0103ba0 0x7f58e006f4f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58c800e9a0 tx=0x7f58c800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:20.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.668+0000 7f58dcff9700 1 -- 172.21.15.57:0/1411044751 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8009790 con 0x7f58e0103ba0 2023-07-20T18:58:20.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.668+0000 7f58dcff9700 1 -- 172.21.15.57:0/1411044751 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f58c8010430 con 0x7f58e0103ba0 2023-07-20T18:58:20.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.668+0000 7f58e55ff700 1 -- 172.21.15.57:0/1411044751 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d00095f0 con 0x7f58e0103ba0 2023-07-20T18:58:20.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.668+0000 7f58dcff9700 1 -- 172.21.15.57:0/1411044751 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8018680 con 0x7f58e0103ba0 2023-07-20T18:58:20.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.669+0000 7f58e55ff700 1 -- 172.21.15.57:0/1411044751 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e01a71b0 con 0x7f58e0103ba0 2023-07-20T18:58:20.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.670+0000 7f58e55ff700 1 -- 172.21.15.57:0/1411044751 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e00fc720 con 0x7f58e0103ba0 2023-07-20T18:58:20.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.672+0000 7f58dcff9700 1 -- 172.21.15.57:0/1411044751 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f58c8010910 con 0x7f58e0103ba0 2023-07-20T18:58:20.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.673+0000 7f58dcff9700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f58cc0a1220 0x7f58cc0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:20.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.673+0000 7f58dcff9700 1 -- 172.21.15.57:0/1411044751 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f58c8014070 con 0x7f58e0103ba0 2023-07-20T18:58:20.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.673+0000 7f58deffd700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f58cc0a1220 0x7f58cc0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:20.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.674+0000 7f58deffd700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f58cc0a1220 0x7f58cc0a36e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f58d000b2e0 tx=0x7f58d0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:20.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.675+0000 7f58dcff9700 1 -- 172.21.15.57:0/1411044751 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f58c8096450 con 0x7f58e0103ba0 2023-07-20T18:58:21.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.978+0000 7f58e55ff700 1 -- 172.21.15.57:0/1411044751 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58e002c970 con 0x7f58e0103ba0 2023-07-20T18:58:21.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.979+0000 7f58dcff9700 1 -- 172.21.15.57:0/1411044751 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8076 (secure 0 0 0) 0x7f58c80105a0 con 0x7f58e0103ba0 2023-07-20T18:58:21.028 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:21.028 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/3251478878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":3251478878},{"type":"v1","addr":"172.21.15.57:6837","nonce":3251478878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14502,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1597260444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1597260444},{"type":"v1","addr":"172.21.15.57:6839","nonce":1597260444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2219652585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2219652585},{"type":"v1","addr":"172.21.15.192:6835","nonce":2219652585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1373743783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1373743783},{"type":"v1","addr":"172.21.15.88:6835","nonce":1373743783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2391717255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2391717255},{"type":"v1","addr":"172.21.15.192:6837","nonce":2391717255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/678288704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":678288704},{"type":"v1","addr":"172.21.15.88:6837","nonce":678288704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/1456232015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1456232015},{"type":"v1","addr":"172.21.15.192:6839","nonce":1456232015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T18:58:21.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.981+0000 7f58e55ff700 1 -- 172.21.15.57:0/1411044751 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f58cc0a1220 msgr2=0x7f58cc0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:21.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.981+0000 7f58e55ff700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f58cc0a1220 0x7f58cc0a36e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f58d000b2e0 tx=0x7f58d0005fb0 comp rx=0 tx=0).stop 2023-07-20T18:58:21.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.981+0000 7f58e55ff700 1 -- 172.21.15.57:0/1411044751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58e0103ba0 msgr2=0x7f58e006f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:21.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.981+0000 7f58e55ff700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58e0103ba0 0x7f58e006f4f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58c800e9a0 tx=0x7f58c800ecb0 comp rx=0 tx=0).stop 2023-07-20T18:58:21.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.982+0000 7f58e55ff700 1 -- 172.21.15.57:0/1411044751 shutdown_connections 2023-07-20T18:58:21.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.982+0000 7f58e55ff700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f58cc0a1220 0x7f58cc0a36e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:21.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.982+0000 7f58e55ff700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e01064d0 0x7f58e01a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:21.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.982+0000 7f58e55ff700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0101270 0x7f58e006efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:21.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.982+0000 7f58e55ff700 1 --2- 172.21.15.57:0/1411044751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58e0103ba0 0x7f58e006f4f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:21.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.982+0000 7f58e55ff700 1 -- 172.21.15.57:0/1411044751 >> 172.21.15.57:0/1411044751 conn(0x7f58e00faba0 msgr2=0x7f58e0101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:21.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.982+0000 7f58e55ff700 1 -- 172.21.15.57:0/1411044751 shutdown_connections 2023-07-20T18:58:21.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:20.982+0000 7f58e55ff700 1 -- 172.21.15.57:0/1411044751 wait complete. 2023-07-20T18:58:21.032 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 11 2023-07-20T18:58:21.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:21.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:21.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:21.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:21.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:21.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:21.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: Deploying daemon mds.a on smithi057 2023-07-20T18:58:21.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:21.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.57:6838/1597260444,v1:172.21.15.57:6839/1597260444] up:boot 2023-07-20T18:58:21.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: fsmap 10 up:standby 2023-07-20T18:58:21.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T18:58:21.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:21 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1411044751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:21.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:21.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:21.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:21.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:21.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:21.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:21.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: Deploying daemon mds.a on smithi057 2023-07-20T18:58:21.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:21.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.57:6838/1597260444,v1:172.21.15.57:6839/1597260444] up:boot 2023-07-20T18:58:21.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: fsmap 10 up:standby 2023-07-20T18:58:21.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T18:58:21.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:21 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1411044751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:21.429 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:21.429 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:21.429 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:21.429 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:21.429 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:21.430 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:21.430 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: Deploying daemon mds.a on smithi057 2023-07-20T18:58:21.430 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:21.430 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.57:6838/1597260444,v1:172.21.15.57:6839/1597260444] up:boot 2023-07-20T18:58:21.430 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: fsmap 10 up:standby 2023-07-20T18:58:21.430 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T18:58:21.430 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:21 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1411044751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:22.735 INFO:journalctl@ceph.mds.a.smithi057.stdout:Jul 20 18:58:22 smithi057 systemd[1]: Starting Ceph mds.a for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:23.051 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:23.159 INFO:journalctl@ceph.mds.a.smithi057.stdout:Jul 20 18:58:22 smithi057 podman[161158]: 2023-07-20T18:58:23.159 INFO:journalctl@ceph.mds.a.smithi057.stdout:Jul 20 18:58:23 smithi057 ceph-mds[161217]: -- 172.21.15.57:0/3496069751 <== mon.0 v2:172.21.15.57:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562e04dc71e0 con 0x562e04d9cc00 2023-07-20T18:58:23.160 INFO:journalctl@ceph.mds.a.smithi057.stdout:Jul 20 18:58:23 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-a[161174]: starting mds.a at 2023-07-20T18:58:23.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.220+0000 7f53a4dff700 1 -- 172.21.15.57:0/711578870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a0101960 msgr2=0x7f53a0101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:23.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.220+0000 7f53a4dff700 1 --2- 172.21.15.57:0/711578870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a0101960 0x7f53a0101dc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f53880099c0 tx=0x7f5388009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.220+0000 7f53a4dff700 1 -- 172.21.15.57:0/711578870 shutdown_connections 2023-07-20T18:58:23.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.220+0000 7f53a4dff700 1 --2- 172.21.15.57:0/711578870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53a0100770 0x7f53a0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.220+0000 7f53a4dff700 1 --2- 172.21.15.57:0/711578870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53a0102300 0x7f53a0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.220+0000 7f53a4dff700 1 --2- 172.21.15.57:0/711578870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a0101960 0x7f53a0101dc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.220+0000 7f53a4dff700 1 -- 172.21.15.57:0/711578870 >> 172.21.15.57:0/711578870 conn(0x7f53a00fbca0 msgr2=0x7f53a00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:23.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.220+0000 7f53a4dff700 1 -- 172.21.15.57:0/711578870 shutdown_connections 2023-07-20T18:58:23.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.220+0000 7f53a4dff700 1 -- 172.21.15.57:0/711578870 wait complete. 2023-07-20T18:58:23.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.221+0000 7f53a4dff700 1 Processor -- start 2023-07-20T18:58:23.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.221+0000 7f53a4dff700 1 -- start start 2023-07-20T18:58:23.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f53a4dff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53a0100770 0x7f53a0074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:23.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f53a4dff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53a0101960 0x7f53a0072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:23.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f53a4dff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a0102300 0x7f53a00730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:23.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f53a4dff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a0073840 con 0x7f53a0100770 2023-07-20T18:58:23.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f53a4dff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a00739b0 con 0x7f53a0102300 2023-07-20T18:58:23.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f53a4dff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a01a0560 con 0x7f53a0101960 2023-07-20T18:58:23.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f539e59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53a0100770 0x7f53a0074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:23.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f539e59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53a0100770 0x7f53a0074520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44234/0 (socket says 172.21.15.57:44234) 2023-07-20T18:58:23.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f539e59c700 1 -- 172.21.15.57:0/751894630 learned_addr learned my addr 172.21.15.57:0/751894630 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:23.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f5397fff700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53a0101960 0x7f53a0072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:23.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.222+0000 7f539ed9d700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a0102300 0x7f53a00730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:23.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f539e59c700 1 -- 172.21.15.57:0/751894630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53a0101960 msgr2=0x7f53a0072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:23.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f539e59c700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53a0101960 0x7f53a0072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f539e59c700 1 -- 172.21.15.57:0/751894630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a0102300 msgr2=0x7f53a00730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:23.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f539e59c700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a0102300 0x7f53a00730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f539e59c700 1 -- 172.21.15.57:0/751894630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880095f0 con 0x7f53a0100770 2023-07-20T18:58:23.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f539e59c700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53a0100770 0x7f53a0074520 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f539000d980 tx=0x7f539000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:23.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f53977fe700 1 -- 172.21.15.57:0/751894630 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390004020 con 0x7f53a0100770 2023-07-20T18:58:23.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f53977fe700 1 -- 172.21.15.57:0/751894630 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f539000f690 con 0x7f53a0100770 2023-07-20T18:58:23.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f53a4dff700 1 -- 172.21.15.57:0/751894630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a01a0840 con 0x7f53a0100770 2023-07-20T18:58:23.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f53977fe700 1 -- 172.21.15.57:0/751894630 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390003b40 con 0x7f53a0100770 2023-07-20T18:58:23.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.223+0000 7f53a4dff700 1 -- 172.21.15.57:0/751894630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a01a0d50 con 0x7f53a0100770 2023-07-20T18:58:23.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.224+0000 7f53a4dff700 1 -- 172.21.15.57:0/751894630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a0107650 con 0x7f53a0100770 2023-07-20T18:58:23.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.226+0000 7f53977fe700 1 -- 172.21.15.57:0/751894630 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f539000fb20 con 0x7f53a0100770 2023-07-20T18:58:23.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.229+0000 7f53977fe700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f538c0a11a0 0x7f538c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:23.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.229+0000 7f53977fe700 1 -- 172.21.15.57:0/751894630 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f53900cc0e0 con 0x7f53a0100770 2023-07-20T18:58:23.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.229+0000 7f5397fff700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f538c0a11a0 0x7f538c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:23.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.229+0000 7f5397fff700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f538c0a11a0 0x7f538c0a3660 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f538800b580 tx=0x7f53880058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.233+0000 7f53977fe700 1 -- 172.21.15.57:0/751894630 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5390095c30 con 0x7f53a0100770 2023-07-20T18:58:23.484 INFO:journalctl@ceph.mds.a.smithi057.stdout:Jul 20 18:58:23 smithi057 bash[161158]: 7237e133190e22b3da8f5bb628195bb5bc82478c7a47432e45a72964efa4111a 2023-07-20T18:58:23.485 INFO:journalctl@ceph.mds.a.smithi057.stdout:Jul 20 18:58:23 smithi057 systemd[1]: Started Ceph mds.a for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:23.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.490+0000 7f53a4dff700 1 -- 172.21.15.57:0/751894630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53a002c970 con 0x7f53a0100770 2023-07-20T18:58:23.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.490+0000 7f53977fe700 1 -- 172.21.15.57:0/751894630 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8076 (secure 0 0 0) 0x7f5390099250 con 0x7f53a0100770 2023-07-20T18:58:23.539 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:23.540 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/3251478878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":3251478878},{"type":"v1","addr":"172.21.15.57:6837","nonce":3251478878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14502,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1597260444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1597260444},{"type":"v1","addr":"172.21.15.57:6839","nonce":1597260444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2219652585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2219652585},{"type":"v1","addr":"172.21.15.192:6835","nonce":2219652585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1373743783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1373743783},{"type":"v1","addr":"172.21.15.88:6835","nonce":1373743783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2391717255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2391717255},{"type":"v1","addr":"172.21.15.192:6837","nonce":2391717255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/678288704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":678288704},{"type":"v1","addr":"172.21.15.88:6837","nonce":678288704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/1456232015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1456232015},{"type":"v1","addr":"172.21.15.192:6839","nonce":1456232015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T18:58:23.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 -- 172.21.15.57:0/751894630 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f538c0a11a0 msgr2=0x7f538c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:23.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f538c0a11a0 0x7f538c0a3660 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f538800b580 tx=0x7f53880058e0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 -- 172.21.15.57:0/751894630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53a0100770 msgr2=0x7f53a0074520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:23.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53a0100770 0x7f53a0074520 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f539000d980 tx=0x7f539000dc90 comp rx=0 tx=0).stop 2023-07-20T18:58:23.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 -- 172.21.15.57:0/751894630 shutdown_connections 2023-07-20T18:58:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f538c0a11a0 0x7f538c0a3660 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a0102300 0x7f53a00730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53a0100770 0x7f53a0074520 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 --2- 172.21.15.57:0/751894630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53a0101960 0x7f53a0072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 -- 172.21.15.57:0/751894630 >> 172.21.15.57:0/751894630 conn(0x7f53a00fbca0 msgr2=0x7f53a00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 -- 172.21.15.57:0/751894630 shutdown_connections 2023-07-20T18:58:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:23.493+0000 7f53a4dff700 1 -- 172.21.15.57:0/751894630 wait complete. 2023-07-20T18:58:23.544 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 11 2023-07-20T18:58:23.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:23 smithi057 ceph-mon[126340]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:23.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:23.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:23.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:23.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:23.986 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:23.986 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:23 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:23.986 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:23 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/751894630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:24.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:23 smithi192 ceph-mon[129200]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:24.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:24.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:24.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:24.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:24.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:24.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:23 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:24.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:23 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/751894630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:24.101 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:23 smithi088 ceph-mon[128823]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:24.102 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:24.102 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:24.102 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:24.102 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T18:58:24.102 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T18:58:24.102 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:23 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:24.102 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:23 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/751894630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:24.792 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 18:58:24 smithi088 systemd[1]: Starting Ceph mds.b for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:58:25.081 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:24 smithi088 ceph-mon[128823]: Deploying daemon mds.b on smithi088 2023-07-20T18:58:25.081 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:24 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.57:6840/1188419010,v1:172.21.15.57:6841/1188419010] up:boot 2023-07-20T18:58:25.081 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:24 smithi088 ceph-mon[128823]: fsmap 11 up:standby 2023-07-20T18:58:25.081 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:24 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T18:58:25.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:24 smithi192 ceph-mon[129200]: Deploying daemon mds.b on smithi088 2023-07-20T18:58:25.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:24 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.57:6840/1188419010,v1:172.21.15.57:6841/1188419010] up:boot 2023-07-20T18:58:25.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:24 smithi192 ceph-mon[129200]: fsmap 11 up:standby 2023-07-20T18:58:25.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:24 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T18:58:25.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:24 smithi057 ceph-mon[126340]: Deploying daemon mds.b on smithi088 2023-07-20T18:58:25.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:24 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.57:6840/1188419010,v1:172.21.15.57:6841/1188419010] up:boot 2023-07-20T18:58:25.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:24 smithi057 ceph-mon[126340]: fsmap 11 up:standby 2023-07-20T18:58:25.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:24 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T18:58:25.346 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 18:58:25 smithi088 podman[154159]: 2023-07-20T18:58:25.563 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:25.632 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 18:58:25 smithi088 ceph-mds[154199]: -- 172.21.15.88:0/1793286079 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55af243791e0 con 0x55af2434e800 2023-07-20T18:58:25.632 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 18:58:25 smithi088 bash[154159]: 7f0a0e012d7a7301c0d17593026320edeb09911efd439b42fe6b0203f42f5057 2023-07-20T18:58:25.632 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 18:58:25 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-b[154175]: starting mds.b at 2023-07-20T18:58:25.632 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 18:58:25 smithi088 systemd[1]: Started Ceph mds.b for 38a58c3c-272e-11ee-9b34-001a4aab830c. 2023-07-20T18:58:25.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.723+0000 7fd6083ff700 1 -- 172.21.15.57:0/2890600711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd600101960 msgr2=0x7fd600101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:25.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.723+0000 7fd6083ff700 1 --2- 172.21.15.57:0/2890600711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd600101960 0x7fd600101dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5f00099c0 tx=0x7fd5f0009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:25.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.723+0000 7fd6083ff700 1 -- 172.21.15.57:0/2890600711 shutdown_connections 2023-07-20T18:58:25.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.723+0000 7fd6083ff700 1 --2- 172.21.15.57:0/2890600711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd600102300 0x7fd600108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:25.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.723+0000 7fd6083ff700 1 --2- 172.21.15.57:0/2890600711 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd600100770 0x7fd600100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:25.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.723+0000 7fd6083ff700 1 --2- 172.21.15.57:0/2890600711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd600101960 0x7fd600101dc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:25.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.723+0000 7fd6083ff700 1 -- 172.21.15.57:0/2890600711 >> 172.21.15.57:0/2890600711 conn(0x7fd6000fbca0 msgr2=0x7fd6000fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:25.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.723+0000 7fd6083ff700 1 -- 172.21.15.57:0/2890600711 shutdown_connections 2023-07-20T18:58:25.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.723+0000 7fd6083ff700 1 -- 172.21.15.57:0/2890600711 wait complete. 2023-07-20T18:58:25.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.724+0000 7fd6083ff700 1 Processor -- start 2023-07-20T18:58:25.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.725+0000 7fd6083ff700 1 -- start start 2023-07-20T18:58:25.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.725+0000 7fd6083ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd600100770 0x7fd600074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:25.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.725+0000 7fd6083ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd600101960 0x7fd600072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:25.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.725+0000 7fd6083ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd600102300 0x7fd6000730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:25.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.725+0000 7fd6083ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd600073840 con 0x7fd600101960 2023-07-20T18:58:25.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.725+0000 7fd6083ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6000739b0 con 0x7fd600102300 2023-07-20T18:58:25.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.725+0000 7fd6083ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6001a0580 con 0x7fd600100770 2023-07-20T18:58:25.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.725+0000 7fd60599a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd600101960 0x7fd600072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:25.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60619b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd600100770 0x7fd600074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:25.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60599a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd600101960 0x7fd600072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56654/0 (socket says 172.21.15.57:56654) 2023-07-20T18:58:25.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60599a700 1 -- 172.21.15.57:0/559824471 learned_addr learned my addr 172.21.15.57:0/559824471 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:25.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60699c700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd600102300 0x7fd6000730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:25.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60599a700 1 -- 172.21.15.57:0/559824471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd600100770 msgr2=0x7fd600074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:25.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60599a700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd600100770 0x7fd600074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:25.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60599a700 1 -- 172.21.15.57:0/559824471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd600102300 msgr2=0x7fd6000730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:25.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60599a700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd600102300 0x7fd6000730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:25.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60599a700 1 -- 172.21.15.57:0/559824471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f00095f0 con 0x7fd600101960 2023-07-20T18:58:25.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60699c700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd600102300 0x7fd6000730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:25.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.726+0000 7fd60599a700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd600101960 0x7fd600072b60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000b580 tx=0x7fd5f000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:25.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.727+0000 7fd5f77fe700 1 -- 172.21.15.57:0/559824471 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f001d070 con 0x7fd600101960 2023-07-20T18:58:25.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.727+0000 7fd6083ff700 1 -- 172.21.15.57:0/559824471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6001a0800 con 0x7fd600101960 2023-07-20T18:58:25.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.727+0000 7fd5f77fe700 1 -- 172.21.15.57:0/559824471 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fd5f0005030 con 0x7fd600101960 2023-07-20T18:58:25.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.727+0000 7fd5f77fe700 1 -- 172.21.15.57:0/559824471 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f00049f0 con 0x7fd600101960 2023-07-20T18:58:25.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.727+0000 7fd6083ff700 1 -- 172.21.15.57:0/559824471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6001a0c80 con 0x7fd600101960 2023-07-20T18:58:25.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.728+0000 7fd6083ff700 1 -- 172.21.15.57:0/559824471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd600107c60 con 0x7fd600101960 2023-07-20T18:58:25.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.731+0000 7fd5f77fe700 1 -- 172.21.15.57:0/559824471 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd5f0005510 con 0x7fd600101960 2023-07-20T18:58:25.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.734+0000 7fd5f77fe700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd5ec0a11a0 0x7fd5ec0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:25.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.734+0000 7fd5f77fe700 1 -- 172.21.15.57:0/559824471 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd5f00cd320 con 0x7fd600101960 2023-07-20T18:58:25.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.734+0000 7fd60619b700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd5ec0a11a0 0x7fd5ec0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:25.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.734+0000 7fd60619b700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd5ec0a11a0 0x7fd5ec0a3660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd6001017c0 tx=0x7fd5fc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:25.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:25.738+0000 7fd5f77fe700 1 -- 172.21.15.57:0/559824471 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd5f0096f00 con 0x7fd600101960 2023-07-20T18:58:25.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:25 smithi057 ceph-mon[126340]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:25.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:25.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:25.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:25.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:25.986 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:25.986 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:25 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:58:26.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.012+0000 7fd6083ff700 1 -- 172.21.15.57:0/559824471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd600061380 con 0x7fd600101960 2023-07-20T18:58:26.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.012+0000 7fd5f77fe700 1 -- 172.21.15.57:0/559824471 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9592 (secure 0 0 0) 0x7fd5f00051a0 con 0x7fd600101960 2023-07-20T18:58:26.061 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:26.062 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/3251478878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":3251478878},{"type":"v1","addr":"172.21.15.57:6837","nonce":3251478878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14502,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1597260444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1597260444},{"type":"v1","addr":"172.21.15.57:6839","nonce":1597260444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/1188419010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":1188419010},{"type":"v1","addr":"172.21.15.57:6841","nonce":1188419010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2219652585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2219652585},{"type":"v1","addr":"172.21.15.192:6835","nonce":2219652585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1373743783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1373743783},{"type":"v1","addr":"172.21.15.88:6835","nonce":1373743783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2391717255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2391717255},{"type":"v1","addr":"172.21.15.192:6837","nonce":2391717255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/678288704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":678288704},{"type":"v1","addr":"172.21.15.88:6837","nonce":678288704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/1456232015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1456232015},{"type":"v1","addr":"172.21.15.192:6839","nonce":1456232015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2098157844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2098157844},{"type":"v1","addr":"172.21.15.88:6839","nonce":2098157844}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-20T18:58:26.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 -- 172.21.15.57:0/559824471 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd5ec0a11a0 msgr2=0x7fd5ec0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd5ec0a11a0 0x7fd5ec0a3660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd6001017c0 tx=0x7fd5fc009260 comp rx=0 tx=0).stop 2023-07-20T18:58:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 -- 172.21.15.57:0/559824471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd600101960 msgr2=0x7fd600072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd600101960 0x7fd600072b60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000b580 tx=0x7fd5f000bfd0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 -- 172.21.15.57:0/559824471 shutdown_connections 2023-07-20T18:58:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd5ec0a11a0 0x7fd5ec0a3660 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd600100770 0x7fd600074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd600102300 0x7fd6000730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 --2- 172.21.15.57:0/559824471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd600101960 0x7fd600072b60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 -- 172.21.15.57:0/559824471 >> 172.21.15.57:0/559824471 conn(0x7fd6000fbca0 msgr2=0x7fd600104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:26.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 -- 172.21.15.57:0/559824471 shutdown_connections 2023-07-20T18:58:26.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.015+0000 7fd6083ff700 1 -- 172.21.15.57:0/559824471 wait complete. 2023-07-20T18:58:26.066 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 13 2023-07-20T18:58:26.083 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T18:58:26.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:25 smithi192 ceph-mon[129200]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:26.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:26.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:26.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:26.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:26.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:26.091 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:25 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:58:26.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:25 smithi088 ceph-mon[128823]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:26.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:26.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:26.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:26.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:26.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:26.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:25 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:58:26.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.244+0000 7fa2e29ff700 1 -- 172.21.15.57:0/900148250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2dc103b60 msgr2=0x7fa2dc103fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.244+0000 7fa2e29ff700 1 --2- 172.21.15.57:0/900148250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2dc103b60 0x7fa2dc103fc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa2d00099c0 tx=0x7fa2d0009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.244+0000 7fa2e29ff700 1 -- 172.21.15.57:0/900148250 shutdown_connections 2023-07-20T18:58:26.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.244+0000 7fa2e29ff700 1 --2- 172.21.15.57:0/900148250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2dc104500 0x7fa2dc10adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.244+0000 7fa2e29ff700 1 --2- 172.21.15.57:0/900148250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2dc102970 0x7fa2dc102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.244+0000 7fa2e29ff700 1 --2- 172.21.15.57:0/900148250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2dc103b60 0x7fa2dc103fc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.244+0000 7fa2e29ff700 1 -- 172.21.15.57:0/900148250 >> 172.21.15.57:0/900148250 conn(0x7fa2dc0fdec0 msgr2=0x7fa2dc100320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:26.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.244+0000 7fa2e29ff700 1 -- 172.21.15.57:0/900148250 shutdown_connections 2023-07-20T18:58:26.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.245+0000 7fa2e29ff700 1 -- 172.21.15.57:0/900148250 wait complete. 2023-07-20T18:58:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.245+0000 7fa2e29ff700 1 Processor -- start 2023-07-20T18:58:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2e29ff700 1 -- start start 2023-07-20T18:58:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2e29ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2dc102970 0x7fa2dc1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2e29ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2dc103b60 0x7fa2dc198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2e29ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2dc104500 0x7fa2dc19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2e29ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc1a0230 con 0x7fa2dc104500 2023-07-20T18:58:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2e29ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc1a0370 con 0x7fa2dc102970 2023-07-20T18:58:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2e29ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc1a0500 con 0x7fa2dc103b60 2023-07-20T18:58:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2db7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2dc103b60 0x7fa2dc198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2db7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2dc103b60 0x7fa2dc198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:33742/0 (socket says 172.21.15.57:33742) 2023-07-20T18:58:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2db7fe700 1 -- 172.21.15.57:0/1610746889 learned_addr learned my addr 172.21.15.57:0/1610746889 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2e0f9c700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2dc104500 0x7fa2dc19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.246+0000 7fa2dbfff700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2dc102970 0x7fa2dc1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.247+0000 7fa2db7fe700 1 -- 172.21.15.57:0/1610746889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2dc102970 msgr2=0x7fa2dc1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.247+0000 7fa2db7fe700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2dc102970 0x7fa2dc1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.247+0000 7fa2db7fe700 1 -- 172.21.15.57:0/1610746889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2dc104500 msgr2=0x7fa2dc19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.247+0000 7fa2db7fe700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2dc104500 0x7fa2dc19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.247+0000 7fa2db7fe700 1 -- 172.21.15.57:0/1610746889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d00095f0 con 0x7fa2dc103b60 2023-07-20T18:58:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.247+0000 7fa2db7fe700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2dc103b60 0x7fa2dc198bf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009990 tx=0x7fa2d000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.247+0000 7fa2d97fa700 1 -- 172.21.15.57:0/1610746889 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d001d070 con 0x7fa2dc103b60 2023-07-20T18:58:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.247+0000 7fa2e29ff700 1 -- 172.21.15.57:0/1610746889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2dc1a0780 con 0x7fa2dc103b60 2023-07-20T18:58:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.247+0000 7fa2d97fa700 1 -- 172.21.15.57:0/1610746889 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fa2d0005050 con 0x7fa2dc103b60 2023-07-20T18:58:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.247+0000 7fa2d97fa700 1 -- 172.21.15.57:0/1610746889 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0004ad0 con 0x7fa2dc103b60 2023-07-20T18:58:26.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.248+0000 7fa2e29ff700 1 -- 172.21.15.57:0/1610746889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2dc1a0bb0 con 0x7fa2dc103b60 2023-07-20T18:58:26.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.249+0000 7fa2e29ff700 1 -- 172.21.15.57:0/1610746889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2dc10e2c0 con 0x7fa2dc103b60 2023-07-20T18:58:26.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.252+0000 7fa2d97fa700 1 -- 172.21.15.57:0/1610746889 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa2d0005530 con 0x7fa2dc103b60 2023-07-20T18:58:26.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.253+0000 7fa2d97fa700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa2c40a11a0 0x7fa2c40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:26.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.254+0000 7fa2d97fa700 1 -- 172.21.15.57:0/1610746889 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa2d00cd2c0 con 0x7fa2dc103b60 2023-07-20T18:58:26.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.254+0000 7fa2dbfff700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa2c40a11a0 0x7fa2c40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:26.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.254+0000 7fa2d97fa700 1 -- 172.21.15.57:0/1610746889 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa2d009a9d0 con 0x7fa2dc103b60 2023-07-20T18:58:26.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.254+0000 7fa2dbfff700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa2c40a11a0 0x7fa2c40a3660 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc1039c0 tx=0x7fa2cc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:26.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.471+0000 7fa2e29ff700 1 -- 172.21.15.57:0/1610746889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2dc109800 con 0x7fa2dc103b60 2023-07-20T18:58:26.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.473+0000 7fa2d97fa700 1 -- 172.21.15.57:0/1610746889 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18373 (secure 0 0 0) 0x7fa2d009a560 con 0x7fa2dc103b60 2023-07-20T18:58:26.522 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:26.523 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":78,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","created":"2023-07-20T18:51:01.245641+0000","modified":"2023-07-20T18:57:45.836366+0000","last_up_change":"2023-07-20T18:57:44.823544+0000","last_in_change":"2023-07-20T18:57:27.428782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T18:54:23.215777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"627a67ea-1366-4b3f-babd-067f75bad1e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6803","nonce":3023441607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6805","nonce":3023441607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6809","nonce":3023441607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6807","nonce":3023441607}]},"public_addr":"172.21.15.57:6803/3023441607","cluster_addr":"172.21.15.57:6805/3023441607","heartbeat_back_addr":"172.21.15.57:6809/3023441607","heartbeat_front_addr":"172.21.15.57:6807/3023441607","state":["exists","up"]},{"osd":1,"uuid":"adcdb236-334e-4478-98a8-7d0628492082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6801","nonce":1809528555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6803","nonce":1809528555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6807","nonce":1809528555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6805","nonce":1809528555}]},"public_addr":"172.21.15.88:6801/1809528555","cluster_addr":"172.21.15.88:6803/1809528555","heartbeat_back_addr":"172.21.15.88:6807/1809528555","heartbeat_front_addr":"172.21.15.88:6805/1809528555","state":["exists","up"]},{"osd":2,"uuid":"3cd56229-2067-433a-b9f9-f47b1b878b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6801","nonce":3891687382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6803","nonce":3891687382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6807","nonce":3891687382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6805","nonce":3891687382}]},"public_addr":"172.21.15.192:6801/3891687382","cluster_addr":"172.21.15.192:6803/3891687382","heartbeat_back_addr":"172.21.15.192:6807/3891687382","heartbeat_front_addr":"172.21.15.192:6805/3891687382","state":["exists","up"]},{"osd":3,"uuid":"aae250ab-e242-4ef8-9c3a-a4fa36102c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6811","nonce":2920717314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6813","nonce":2920717314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6817","nonce":2920717314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6815","nonce":2920717314}]},"public_addr":"172.21.15.57:6811/2920717314","cluster_addr":"172.21.15.57:6813/2920717314","heartbeat_back_addr":"172.21.15.57:6817/2920717314","heartbeat_front_addr":"172.21.15.57:6815/2920717314","state":["exists","up"]},{"osd":4,"uuid":"8c8d1dfa-1cff-4f0a-ae72-22bd522ed696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6809","nonce":3911525610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6811","nonce":3911525610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6815","nonce":3911525610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6813","nonce":3911525610}]},"public_addr":"172.21.15.88:6809/3911525610","cluster_addr":"172.21.15.88:6811/3911525610","heartbeat_back_addr":"172.21.15.88:6815/3911525610","heartbeat_front_addr":"172.21.15.88:6813/3911525610","state":["exists","up"]},{"osd":5,"uuid":"5a2a56ab-2af9-4176-ba4d-b2138f0e7c75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6809","nonce":905436397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6811","nonce":905436397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6815","nonce":905436397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6813","nonce":905436397}]},"public_addr":"172.21.15.192:6809/905436397","cluster_addr":"172.21.15.192:6811/905436397","heartbeat_back_addr":"172.21.15.192:6815/905436397","heartbeat_front_addr":"172.21.15.192:6813/905436397","state":["exists","up"]},{"osd":6,"uuid":"03eafdda-278c-443c-b376-677eeb50262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6819","nonce":333894423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6821","nonce":333894423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6825","nonce":333894423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6823","nonce":333894423}]},"public_addr":"172.21.15.57:6819/333894423","cluster_addr":"172.21.15.57:6821/333894423","heartbeat_back_addr":"172.21.15.57:6825/333894423","heartbeat_front_addr":"172.21.15.57:6823/333894423","state":["exists","up"]},{"osd":7,"uuid":"3f01254e-be73-41c8-b979-3b6a7534e2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6817","nonce":3031485551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6819","nonce":3031485551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6823","nonce":3031485551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6821","nonce":3031485551}]},"public_addr":"172.21.15.88:6817/3031485551","cluster_addr":"172.21.15.88:6819/3031485551","heartbeat_back_addr":"172.21.15.88:6823/3031485551","heartbeat_front_addr":"172.21.15.88:6821/3031485551","state":["exists","up"]},{"osd":8,"uuid":"624a150e-f47b-4a76-8785-c818dd65342e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6817","nonce":918056390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6819","nonce":918056390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6823","nonce":918056390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6821","nonce":918056390}]},"public_addr":"172.21.15.192:6817/918056390","cluster_addr":"172.21.15.192:6819/918056390","heartbeat_back_addr":"172.21.15.192:6823/918056390","heartbeat_front_addr":"172.21.15.192:6821/918056390","state":["exists","up"]},{"osd":9,"uuid":"6d9cb69d-ed7b-4009-9ec5-027becc577d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6827","nonce":1773340763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6829","nonce":1773340763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6833","nonce":1773340763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6831","nonce":1773340763}]},"public_addr":"172.21.15.57:6827/1773340763","cluster_addr":"172.21.15.57:6829/1773340763","heartbeat_back_addr":"172.21.15.57:6833/1773340763","heartbeat_front_addr":"172.21.15.57:6831/1773340763","state":["exists","up"]},{"osd":10,"uuid":"d5799219-0979-40b1-a596-acd37f37a72d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6825","nonce":2781975579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6827","nonce":2781975579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6831","nonce":2781975579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6829","nonce":2781975579}]},"public_addr":"172.21.15.88:6825/2781975579","cluster_addr":"172.21.15.88:6827/2781975579","heartbeat_back_addr":"172.21.15.88:6831/2781975579","heartbeat_front_addr":"172.21.15.88:6829/2781975579","state":["exists","up"]},{"osd":11,"uuid":"e77cac6c-107e-41f9-8efd-e1140c4e3bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6825","nonce":2864190031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6827","nonce":2864190031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6831","nonce":2864190031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6829","nonce":2864190031}]},"public_addr":"172.21.15.192:6825/2864190031","cluster_addr":"172.21.15.192:6827/2864190031","heartbeat_back_addr":"172.21.15.192:6831/2864190031","heartbeat_front_addr":"172.21.15.192:6829/2864190031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:53:38.180971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:00.253039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:20.776122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:48.631094+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:09.288231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:29.595802+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:55.089126+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:15.637849+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:35.985101+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:01.544767+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:22.351743+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:43.255157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6801/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/3804015220":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6800/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1582727629":"2023-07-21T18:51:34.614904+0000","172.21.15.57:6800/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/2044626900":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/748511752":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1151100650":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6801/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/3446341201":"2023-07-21T18:52:02.610729+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T18:58:26.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 -- 172.21.15.57:0/1610746889 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa2c40a11a0 msgr2=0x7fa2c40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa2c40a11a0 0x7fa2c40a3660 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc1039c0 tx=0x7fa2cc00b410 comp rx=0 tx=0).stop 2023-07-20T18:58:26.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 -- 172.21.15.57:0/1610746889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2dc103b60 msgr2=0x7fa2dc198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2dc103b60 0x7fa2dc198bf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009990 tx=0x7fa2d000f690 comp rx=0 tx=0).stop 2023-07-20T18:58:26.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 -- 172.21.15.57:0/1610746889 shutdown_connections 2023-07-20T18:58:26.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa2c40a11a0 0x7fa2c40a3660 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2dc104500 0x7fa2dc19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2dc102970 0x7fa2dc1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 --2- 172.21.15.57:0/1610746889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2dc103b60 0x7fa2dc198bf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 -- 172.21.15.57:0/1610746889 >> 172.21.15.57:0/1610746889 conn(0x7fa2dc0fdec0 msgr2=0x7fa2dc100090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:26.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.475+0000 7fa2e29ff700 1 -- 172.21.15.57:0/1610746889 shutdown_connections 2023-07-20T18:58:26.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.476+0000 7fa2e29ff700 1 -- 172.21.15.57:0/1610746889 wait complete. 2023-07-20T18:58:26.544 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T18:54:23.215777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T18:58:26.544 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T18:58:26.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.713+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/829251815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e681022f0 msgr2=0x7f4e68108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.713+0000 7f4e6f1ff700 1 --2- 172.21.15.57:0/829251815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e681022f0 0x7f4e68108ba0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4e60009fc0 tx=0x7f4e6000b330 comp rx=0 tx=0).stop 2023-07-20T18:58:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.713+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/829251815 shutdown_connections 2023-07-20T18:58:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.713+0000 7f4e6f1ff700 1 --2- 172.21.15.57:0/829251815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e68100760 0x7f4e68100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.713+0000 7f4e6f1ff700 1 --2- 172.21.15.57:0/829251815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e681022f0 0x7f4e68108ba0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.713+0000 7f4e6f1ff700 1 --2- 172.21.15.57:0/829251815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e68101950 0x7f4e68101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.713+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/829251815 >> 172.21.15.57:0/829251815 conn(0x7f4e680fbcb0 msgr2=0x7f4e680fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:26.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.713+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/829251815 shutdown_connections 2023-07-20T18:58:26.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.713+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/829251815 wait complete. 2023-07-20T18:58:26.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.714+0000 7f4e6f1ff700 1 Processor -- start 2023-07-20T18:58:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.714+0000 7f4e6f1ff700 1 -- start start 2023-07-20T18:58:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6f1ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e68100760 0x7f4e68074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6f1ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e68101950 0x7f4e68072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6f1ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e681022f0 0x7f4e680730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6f1ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e68073840 con 0x7f4e68101950 2023-07-20T18:58:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6f1ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e680739b0 con 0x7f4e68100760 2023-07-20T18:58:26.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6f1ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e681a06b0 con 0x7f4e681022f0 2023-07-20T18:58:26.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6d79c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e681022f0 0x7f4e680730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:26.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e67fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e68101950 0x7f4e68072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:26.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6d79c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e681022f0 0x7f4e680730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:33760/0 (socket says 172.21.15.57:33760) 2023-07-20T18:58:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e67fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e68101950 0x7f4e68072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56704/0 (socket says 172.21.15.57:56704) 2023-07-20T18:58:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6d79c700 1 -- 172.21.15.57:0/639601947 learned_addr learned my addr 172.21.15.57:0/639601947 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6cf9b700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e68100760 0x7f4e68074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6d79c700 1 -- 172.21.15.57:0/639601947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e68100760 msgr2=0x7f4e68074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.715+0000 7f4e6d79c700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e68100760 0x7f4e68074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e6d79c700 1 -- 172.21.15.57:0/639601947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e68101950 msgr2=0x7f4e68072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e6d79c700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e68101950 0x7f4e68072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e6d79c700 1 -- 172.21.15.57:0/639601947 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e60009c40 con 0x7f4e681022f0 2023-07-20T18:58:26.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e67fff700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e68101950 0x7f4e68072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:26.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e6cf9b700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e68100760 0x7f4e68074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:26.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e6d79c700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e681022f0 0x7f4e680730a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4e60004c60 tx=0x7f4e60004c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:26.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e65ffb700 1 -- 172.21.15.57:0/639601947 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6000e040 con 0x7f4e681022f0 2023-07-20T18:58:26.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e65ffb700 1 -- 172.21.15.57:0/639601947 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f4e6000be00 con 0x7f4e681022f0 2023-07-20T18:58:26.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e65ffb700 1 -- 172.21.15.57:0/639601947 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e600052a0 con 0x7f4e681022f0 2023-07-20T18:58:26.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/639601947 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e681a0930 con 0x7f4e681022f0 2023-07-20T18:58:26.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.716+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/639601947 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e681a0d60 con 0x7f4e681022f0 2023-07-20T18:58:26.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.717+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/639601947 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6810c110 con 0x7f4e681022f0 2023-07-20T18:58:26.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.719+0000 7f4e65ffb700 1 -- 172.21.15.57:0/639601947 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4e6001d470 con 0x7f4e681022f0 2023-07-20T18:58:26.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.720+0000 7f4e65ffb700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4e580a1150 0x7f4e580a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:26.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.721+0000 7f4e65ffb700 1 -- 172.21.15.57:0/639601947 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4e600cd9b0 con 0x7f4e681022f0 2023-07-20T18:58:26.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.721+0000 7f4e6cf9b700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4e580a1150 0x7f4e580a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:26.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.721+0000 7f4e6cf9b700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4e580a1150 0x7f4e580a3610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e681017b0 tx=0x7f4e5c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:26.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.722+0000 7f4e65ffb700 1 -- 172.21.15.57:0/639601947 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4e600976b0 con 0x7f4e681022f0 2023-07-20T18:58:26.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.941+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/639601947 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4e68107570 con 0x7f4e681022f0 2023-07-20T18:58:26.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.942+0000 7f4e65ffb700 1 -- 172.21.15.57:0/639601947 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4e6009acd0 con 0x7f4e681022f0 2023-07-20T18:58:26.991 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-07-20T18:58:26.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/639601947 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4e580a1150 msgr2=0x7f4e580a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4e580a1150 0x7f4e580a3610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e681017b0 tx=0x7f4e5c009260 comp rx=0 tx=0).stop 2023-07-20T18:58:26.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/639601947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e681022f0 msgr2=0x7f4e680730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:26.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e681022f0 0x7f4e680730a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4e60004c60 tx=0x7f4e60004c90 comp rx=0 tx=0).stop 2023-07-20T18:58:26.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/639601947 shutdown_connections 2023-07-20T18:58:26.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4e580a1150 0x7f4e580a3610 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e681022f0 0x7f4e680730a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e68100760 0x7f4e68074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 --2- 172.21.15.57:0/639601947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e68101950 0x7f4e68072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:26.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/639601947 >> 172.21.15.57:0/639601947 conn(0x7f4e680fbcb0 msgr2=0x7f4e680fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:26.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/639601947 shutdown_connections 2023-07-20T18:58:26.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:26.945+0000 7f4e6f1ff700 1 -- 172.21.15.57:0/639601947 wait complete. 2023-07-20T18:58:27.016 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-20T18:58:27.079 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:26 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] up:boot 2023-07-20T18:58:27.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:26 smithi057 ceph-mon[126340]: fsmap 12 up:standby 2023-07-20T18:58:27.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:26 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T18:58:27.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:26 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/559824471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:27.080 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:26 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1610746889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:27.089 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:26 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] up:boot 2023-07-20T18:58:27.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:26 smithi192 ceph-mon[129200]: fsmap 12 up:standby 2023-07-20T18:58:27.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:26 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T18:58:27.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:26 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/559824471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:27.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:26 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1610746889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:27.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:26 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] up:boot 2023-07-20T18:58:27.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:26 smithi088 ceph-mon[128823]: fsmap 12 up:standby 2023-07-20T18:58:27.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:26 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T18:58:27.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:26 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/559824471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:27.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:26 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1610746889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:27.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.185+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/2513660332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c34101960 msgr2=0x7f4c34101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:27.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.185+0000 7f4c3c1ff700 1 --2- 172.21.15.57:0/2513660332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c34101960 0x7f4c34101dc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4c240099c0 tx=0x7f4c24009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.185+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/2513660332 shutdown_connections 2023-07-20T18:58:27.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.185+0000 7f4c3c1ff700 1 --2- 172.21.15.57:0/2513660332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c34100770 0x7f4c34100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.185+0000 7f4c3c1ff700 1 --2- 172.21.15.57:0/2513660332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34102300 0x7f4c34108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.185+0000 7f4c3c1ff700 1 --2- 172.21.15.57:0/2513660332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c34101960 0x7f4c34101dc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.185+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/2513660332 >> 172.21.15.57:0/2513660332 conn(0x7f4c340fbca0 msgr2=0x7f4c340fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:27.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.185+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/2513660332 shutdown_connections 2023-07-20T18:58:27.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.185+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/2513660332 wait complete. 2023-07-20T18:58:27.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.186+0000 7f4c3c1ff700 1 Processor -- start 2023-07-20T18:58:27.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.186+0000 7f4c3c1ff700 1 -- start start 2023-07-20T18:58:27.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3c1ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c34100770 0x7f4c34074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:27.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3c1ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34101960 0x7f4c34072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:27.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3c1ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c34102300 0x7f4c340730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:27.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3c1ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c34073840 con 0x7f4c34102300 2023-07-20T18:58:27.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3c1ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c340739b0 con 0x7f4c34101960 2023-07-20T18:58:27.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3c1ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341a0530 con 0x7f4c34100770 2023-07-20T18:58:27.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3a79c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c34102300 0x7f4c340730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:27.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3979a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34101960 0x7f4c34072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:27.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c39f9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c34100770 0x7f4c34074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:27.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3979a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34101960 0x7f4c34072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:33710/0 (socket says 172.21.15.57:33710) 2023-07-20T18:58:27.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3979a700 1 -- 172.21.15.57:0/1681304090 learned_addr learned my addr 172.21.15.57:0/1681304090 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:27.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3a79c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c34102300 0x7f4c340730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56724/0 (socket says 172.21.15.57:56724) 2023-07-20T18:58:27.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3979a700 1 -- 172.21.15.57:0/1681304090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c34100770 msgr2=0x7f4c34074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:27.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3979a700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c34100770 0x7f4c34074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3979a700 1 -- 172.21.15.57:0/1681304090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c34102300 msgr2=0x7f4c340730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:27.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3979a700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c34102300 0x7f4c340730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c3979a700 1 -- 172.21.15.57:0/1681304090 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c009c40 con 0x7f4c34101960 2023-07-20T18:58:27.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.187+0000 7f4c39f9b700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c34100770 0x7f4c34074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:27.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.188+0000 7f4c3979a700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34101960 0x7f4c34072b60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400f690 tx=0x7f4c2400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:27.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.188+0000 7f4c2affd700 1 -- 172.21.15.57:0/1681304090 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2401d070 con 0x7f4c34101960 2023-07-20T18:58:27.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.188+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/1681304090 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c240095f0 con 0x7f4c34101960 2023-07-20T18:58:27.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.188+0000 7f4c2affd700 1 -- 172.21.15.57:0/1681304090 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f4c2400fb70 con 0x7f4c34101960 2023-07-20T18:58:27.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.188+0000 7f4c2affd700 1 -- 172.21.15.57:0/1681304090 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c240173f0 con 0x7f4c34101960 2023-07-20T18:58:27.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.188+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/1681304090 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c341a0aa0 con 0x7f4c34101960 2023-07-20T18:58:27.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.189+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/1681304090 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3410c0c0 con 0x7f4c34101960 2023-07-20T18:58:27.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.191+0000 7f4c2affd700 1 -- 172.21.15.57:0/1681304090 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4c2400fce0 con 0x7f4c34101960 2023-07-20T18:58:27.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.193+0000 7f4c2affd700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c200a1220 0x7f4c200a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.193+0000 7f4c2affd700 1 -- 172.21.15.57:0/1681304090 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4c240cd5d0 con 0x7f4c34101960 2023-07-20T18:58:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.193+0000 7f4c39f9b700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c200a1220 0x7f4c200a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.194+0000 7f4c39f9b700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c200a1220 0x7f4c200a36e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4c341017c0 tx=0x7f4c30009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.196+0000 7f4c2affd700 1 -- 172.21.15.57:0/1681304090 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4c240971b0 con 0x7f4c34101960 2023-07-20T18:58:27.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.457+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/1681304090 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4c341075d0 con 0x7f4c34101960 2023-07-20T18:58:27.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.458+0000 7f4c2affd700 1 -- 172.21.15.57:0/1681304090 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4c2409a7d0 con 0x7f4c34101960 2023-07-20T18:58:27.507 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:27.508 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-07-20T18:58:27.508 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-07-20T18:58:27.508 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-07-20T18:58:27.508 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-07-20T18:58:27.508 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-07-20T18:58:27.508 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-07-20T18:58:27.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/1681304090 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c200a1220 msgr2=0x7f4c200a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:27.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c200a1220 0x7f4c200a36e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4c341017c0 tx=0x7f4c30009260 comp rx=0 tx=0).stop 2023-07-20T18:58:27.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/1681304090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34101960 msgr2=0x7f4c34072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:27.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34101960 0x7f4c34072b60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400f690 tx=0x7f4c2400bfd0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/1681304090 shutdown_connections 2023-07-20T18:58:27.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c200a1220 0x7f4c200a36e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c34100770 0x7f4c34074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c34102300 0x7f4c340730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 --2- 172.21.15.57:0/1681304090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34101960 0x7f4c34072b60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/1681304090 >> 172.21.15.57:0/1681304090 conn(0x7f4c340fbca0 msgr2=0x7f4c340fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:27.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/1681304090 shutdown_connections 2023-07-20T18:58:27.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.461+0000 7f4c3c1ff700 1 -- 172.21.15.57:0/1681304090 wait complete. 2023-07-20T18:58:27.530 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-20T18:58:27.530 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-20T18:58:27.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.702+0000 7f27b65ff700 1 -- 172.21.15.57:0/3806567074 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b0074750 msgr2=0x7f27b0072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:27.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.702+0000 7f27b65ff700 1 --2- 172.21.15.57:0/3806567074 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b0074750 0x7f27b0072ba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009990 tx=0x7f27a0009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.702+0000 7f27b65ff700 1 -- 172.21.15.57:0/3806567074 shutdown_connections 2023-07-20T18:58:27.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.702+0000 7f27b65ff700 1 --2- 172.21.15.57:0/3806567074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b0108660 0x7f27b010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.702+0000 7f27b65ff700 1 --2- 172.21.15.57:0/3806567074 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b0074750 0x7f27b0072ba0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.702+0000 7f27b65ff700 1 --2- 172.21.15.57:0/3806567074 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b0073170 0x7f27b00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.702+0000 7f27b65ff700 1 -- 172.21.15.57:0/3806567074 >> 172.21.15.57:0/3806567074 conn(0x7f27b00fbd20 msgr2=0x7f27b00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:27.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.702+0000 7f27b65ff700 1 -- 172.21.15.57:0/3806567074 shutdown_connections 2023-07-20T18:58:27.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.702+0000 7f27b65ff700 1 -- 172.21.15.57:0/3806567074 wait complete. 2023-07-20T18:58:27.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.703+0000 7f27b65ff700 1 Processor -- start 2023-07-20T18:58:27.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27b65ff700 1 -- start start 2023-07-20T18:58:27.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27b65ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b0073170 0x7f27b01987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:27.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27b65ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b0074750 0x7f27b0198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:27.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27b65ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b0108660 0x7f27b019fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:27.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27b65ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b01a0320 con 0x7f27b0074750 2023-07-20T18:58:27.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27b65ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b01a0460 con 0x7f27b0108660 2023-07-20T18:58:27.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27b65ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b01a05f0 con 0x7f27b0073170 2023-07-20T18:58:27.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27affff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b0073170 0x7f27b01987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:27.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27af7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b0074750 0x7f27b0198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:27.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27affff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b0073170 0x7f27b01987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:33792/0 (socket says 172.21.15.57:33792) 2023-07-20T18:58:27.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27affff700 1 -- 172.21.15.57:0/729094379 learned_addr learned my addr 172.21.15.57:0/729094379 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:27.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.704+0000 7f27b4b9c700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b0108660 0x7f27b019fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:27.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.705+0000 7f27af7fe700 1 -- 172.21.15.57:0/729094379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b0073170 msgr2=0x7f27b01987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:27.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.705+0000 7f27af7fe700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b0073170 0x7f27b01987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.705+0000 7f27af7fe700 1 -- 172.21.15.57:0/729094379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b0108660 msgr2=0x7f27b019fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:27.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.705+0000 7f27af7fe700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b0108660 0x7f27b019fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.705+0000 7f27af7fe700 1 -- 172.21.15.57:0/729094379 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a00095f0 con 0x7f27b0074750 2023-07-20T18:58:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.705+0000 7f27af7fe700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b0074750 0x7f27b0198ce0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f27a400e9f0 tx=0x7f27a400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.705+0000 7f27ad7fa700 1 -- 172.21.15.57:0/729094379 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a400c9c0 con 0x7f27b0074750 2023-07-20T18:58:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.705+0000 7f27ad7fa700 1 -- 172.21.15.57:0/729094379 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f27a4010430 con 0x7f27b0074750 2023-07-20T18:58:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.705+0000 7f27ad7fa700 1 -- 172.21.15.57:0/729094379 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4018680 con 0x7f27b0074750 2023-07-20T18:58:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.705+0000 7f27b65ff700 1 -- 172.21.15.57:0/729094379 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b01a08d0 con 0x7f27b0074750 2023-07-20T18:58:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.706+0000 7f27b65ff700 1 -- 172.21.15.57:0/729094379 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b01a0d90 con 0x7f27b0074750 2023-07-20T18:58:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.707+0000 7f27b65ff700 1 -- 172.21.15.57:0/729094379 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b0106d60 con 0x7f27b0074750 2023-07-20T18:58:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.708+0000 7f27ad7fa700 1 -- 172.21.15.57:0/729094379 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f27a4010910 con 0x7f27b0074750 2023-07-20T18:58:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.710+0000 7f27ad7fa700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f27980a1220 0x7f27980a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:27.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.710+0000 7f27ad7fa700 1 -- 172.21.15.57:0/729094379 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f27a4014070 con 0x7f27b0074750 2023-07-20T18:58:27.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.710+0000 7f27affff700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f27980a1220 0x7f27980a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:27.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.711+0000 7f27affff700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f27980a1220 0x7f27980a36e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f27a000b6b0 tx=0x7f27a0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:27.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.714+0000 7f27ad7fa700 1 -- 172.21.15.57:0/729094379 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f27a4051780 con 0x7f27b0074750 2023-07-20T18:58:27.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:27.937+0000 7f27b65ff700 1 -- 172.21.15.57:0/729094379 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f27b002c970 con 0x7f27b0074750 2023-07-20T18:58:28.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:27 smithi192 ceph-mon[129200]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:28.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:27 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/639601947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:58:28.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:27 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1681304090' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T18:58:28.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:27 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:28.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:27 smithi088 ceph-mon[128823]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:28.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:27 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/639601947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:58:28.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:27 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1681304090' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T18:58:28.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:27 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:28.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:27 smithi057 ceph-mon[126340]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:28.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:27 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/639601947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:58:28.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:27 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1681304090' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T18:58:28.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:27 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:28.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.786+0000 7f27ad7fa700 1 -- 172.21.15.57:0/729094379 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7f27a4096180 con 0x7f27b0074750 2023-07-20T18:58:28.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.791+0000 7f27b65ff700 1 -- 172.21.15.57:0/729094379 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f27980a1220 msgr2=0x7f27980a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:28.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.791+0000 7f27b65ff700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f27980a1220 0x7f27980a36e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f27a000b6b0 tx=0x7f27a0005fd0 comp rx=0 tx=0).stop 2023-07-20T18:58:28.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.791+0000 7f27b65ff700 1 -- 172.21.15.57:0/729094379 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b0074750 msgr2=0x7f27b0198ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:28.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.791+0000 7f27b65ff700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b0074750 0x7f27b0198ce0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f27a400e9f0 tx=0x7f27a400ed00 comp rx=0 tx=0).stop 2023-07-20T18:58:28.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.791+0000 7f27b65ff700 1 -- 172.21.15.57:0/729094379 shutdown_connections 2023-07-20T18:58:28.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.791+0000 7f27b65ff700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f27980a1220 0x7f27980a36e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:28.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.791+0000 7f27b65ff700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b0073170 0x7f27b01987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:28.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.791+0000 7f27b65ff700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b0108660 0x7f27b019fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:28.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.791+0000 7f27b65ff700 1 --2- 172.21.15.57:0/729094379 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b0074750 0x7f27b0198ce0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:28.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.791+0000 7f27b65ff700 1 -- 172.21.15.57:0/729094379 >> 172.21.15.57:0/729094379 conn(0x7f27b00fbd20 msgr2=0x7f27b00fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:28.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.792+0000 7f27b65ff700 1 -- 172.21.15.57:0/729094379 shutdown_connections 2023-07-20T18:58:28.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:28.792+0000 7f27b65ff700 1 -- 172.21.15.57:0/729094379 wait complete. 2023-07-20T18:58:28.842 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-07-20T18:58:28.860 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-07-20T18:58:29.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.039+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/580017710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a481080d0 msgr2=0x7f6a4810eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:29.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.039+0000 7f6a4e7ff700 1 --2- 172.21.15.57:0/580017710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a481080d0 0x7f6a4810eab0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009990 tx=0x7f6a3c009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:29.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.039+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/580017710 shutdown_connections 2023-07-20T18:58:29.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.039+0000 7f6a4e7ff700 1 --2- 172.21.15.57:0/580017710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a48107770 0x7f6a48107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:29.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.039+0000 7f6a4e7ff700 1 --2- 172.21.15.57:0/580017710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a4810eff0 0x7f6a481113e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:29.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.039+0000 7f6a4e7ff700 1 --2- 172.21.15.57:0/580017710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a481080d0 0x7f6a4810eab0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:29.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.039+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/580017710 >> 172.21.15.57:0/580017710 conn(0x7f6a4806d520 msgr2=0x7f6a4806f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:29.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.039+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/580017710 shutdown_connections 2023-07-20T18:58:29.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.039+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/580017710 wait complete. 2023-07-20T18:58:29.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:28 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/729094379' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T18:58:29.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:28 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:28 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:28 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.090 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:28 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.040+0000 7f6a4e7ff700 1 Processor -- start 2023-07-20T18:58:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.040+0000 7f6a4e7ff700 1 -- start start 2023-07-20T18:58:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.040+0000 7f6a4e7ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a48107770 0x7f6a481a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.040+0000 7f6a4e7ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a481080d0 0x7f6a481a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a4e7ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a4810eff0 0x7f6a481ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a4e7ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a481acf00 con 0x7f6a4810eff0 2023-07-20T18:58:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a4e7ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a481ad040 con 0x7f6a481080d0 2023-07-20T18:58:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a4e7ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a481ad1d0 con 0x7f6a48107770 2023-07-20T18:58:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a477fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a481080d0 0x7f6a481a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a477fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a481080d0 0x7f6a481a58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:33746/0 (socket says 172.21.15.57:33746) 2023-07-20T18:58:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a477fe700 1 -- 172.21.15.57:0/283821544 learned_addr learned my addr 172.21.15.57:0/283821544 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a4cd9c700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a4810eff0 0x7f6a481ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a47fff700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a48107770 0x7f6a481a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:29.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a477fe700 1 -- 172.21.15.57:0/283821544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a48107770 msgr2=0x7f6a481a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:29.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a477fe700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a48107770 0x7f6a481a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:29.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a477fe700 1 -- 172.21.15.57:0/283821544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a4810eff0 msgr2=0x7f6a481ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:29.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a477fe700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a4810eff0 0x7f6a481ac930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:29.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a477fe700 1 -- 172.21.15.57:0/283821544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c0095f0 con 0x7f6a481080d0 2023-07-20T18:58:29.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.041+0000 7f6a477fe700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a481080d0 0x7f6a481a58c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c004730 tx=0x7f6a3c004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:29.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.042+0000 7f6a457fa700 1 -- 172.21.15.57:0/283821544 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c01d070 con 0x7f6a481080d0 2023-07-20T18:58:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.042+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/283821544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a481ad450 con 0x7f6a481080d0 2023-07-20T18:58:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.042+0000 7f6a457fa700 1 -- 172.21.15.57:0/283821544 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f6a3c004cb0 con 0x7f6a481080d0 2023-07-20T18:58:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.042+0000 7f6a457fa700 1 -- 172.21.15.57:0/283821544 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c005460 con 0x7f6a481080d0 2023-07-20T18:58:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.042+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/283821544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a481ad820 con 0x7f6a481080d0 2023-07-20T18:58:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.043+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/283821544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4810c030 con 0x7f6a481080d0 2023-07-20T18:58:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.045+0000 7f6a457fa700 1 -- 172.21.15.57:0/283821544 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6a3c00fb40 con 0x7f6a481080d0 2023-07-20T18:58:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.046+0000 7f6a457fa700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6a300a1150 0x7f6a300a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.046+0000 7f6a457fa700 1 -- 172.21.15.57:0/283821544 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f6a3c0cd820 con 0x7f6a481080d0 2023-07-20T18:58:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.046+0000 7f6a47fff700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6a300a1150 0x7f6a300a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.047+0000 7f6a47fff700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6a300a1150 0x7f6a300a3610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a38009c60 tx=0x7f6a38009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:29.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.050+0000 7f6a457fa700 1 -- 172.21.15.57:0/283821544 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6a3c05a780 con 0x7f6a481080d0 2023-07-20T18:58:29.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:28 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/729094379' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T18:58:29.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:28 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:28 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:28 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:28 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:28 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/729094379' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T18:58:29.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:28 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:28 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:28 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:28 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:29.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.269+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/283821544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f6a4802c970 con 0x7f6a481080d0 2023-07-20T18:58:29.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.810+0000 7f6a457fa700 1 -- 172.21.15.57:0/283821544 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v80) v1 ==== 171+0+0 (secure 0 0 0) 0x7f6a3c097290 con 0x7f6a481080d0 2023-07-20T18:58:29.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.815+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/283821544 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6a300a1150 msgr2=0x7f6a300a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.815+0000 7f6a4e7ff700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6a300a1150 0x7f6a300a3610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a38009c60 tx=0x7f6a38009260 comp rx=0 tx=0).stop 2023-07-20T18:58:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.815+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/283821544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a481080d0 msgr2=0x7f6a481a58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.815+0000 7f6a4e7ff700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a481080d0 0x7f6a481a58c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c004730 tx=0x7f6a3c004760 comp rx=0 tx=0).stop 2023-07-20T18:58:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.816+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/283821544 shutdown_connections 2023-07-20T18:58:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.816+0000 7f6a4e7ff700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6a300a1150 0x7f6a300a3610 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.816+0000 7f6a4e7ff700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a48107770 0x7f6a481a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.816+0000 7f6a4e7ff700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a4810eff0 0x7f6a481ac930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.816+0000 7f6a4e7ff700 1 --2- 172.21.15.57:0/283821544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a481080d0 0x7f6a481a58c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.816+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/283821544 >> 172.21.15.57:0/283821544 conn(0x7f6a4806d520 msgr2=0x7f6a4806e190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.816+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/283821544 shutdown_connections 2023-07-20T18:58:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:29.817+0000 7f6a4e7ff700 1 -- 172.21.15.57:0/283821544 wait complete. 2023-07-20T18:58:29.868 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-07-20T18:58:29.889 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-20T18:58:30.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.050+0000 7f663b7ff700 1 -- 172.21.15.57:0/606218735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6634101950 msgr2=0x7f6634101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:30.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.050+0000 7f663b7ff700 1 --2- 172.21.15.57:0/606218735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6634101950 0x7f6634101db0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6624009990 tx=0x7f6624009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:30.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.050+0000 7f663b7ff700 1 -- 172.21.15.57:0/606218735 shutdown_connections 2023-07-20T18:58:30.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.050+0000 7f663b7ff700 1 --2- 172.21.15.57:0/606218735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66341022f0 0x7f6634108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:30.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.050+0000 7f663b7ff700 1 --2- 172.21.15.57:0/606218735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6634100760 0x7f6634100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:30.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.050+0000 7f663b7ff700 1 --2- 172.21.15.57:0/606218735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6634101950 0x7f6634101db0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:30.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.050+0000 7f663b7ff700 1 -- 172.21.15.57:0/606218735 >> 172.21.15.57:0/606218735 conn(0x7f66340fbcb0 msgr2=0x7f66340fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:30.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.050+0000 7f663b7ff700 1 -- 172.21.15.57:0/606218735 shutdown_connections 2023-07-20T18:58:30.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.050+0000 7f663b7ff700 1 -- 172.21.15.57:0/606218735 wait complete. 2023-07-20T18:58:30.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.051+0000 7f663b7ff700 1 Processor -- start 2023-07-20T18:58:30.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.051+0000 7f663b7ff700 1 -- start start 2023-07-20T18:58:30.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f663b7ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6634100760 0x7f66341986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:30.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f663b7ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6634101950 0x7f6634198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:30.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f663b7ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66341022f0 0x7f663419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:30.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f663b7ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66341a0230 con 0x7f6634101950 2023-07-20T18:58:30.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f663b7ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66341a0370 con 0x7f66341022f0 2023-07-20T18:58:30.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f663b7ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66341a0500 con 0x7f6634100760 2023-07-20T18:58:30.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f6638d9a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6634101950 0x7f6634198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:30.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f6638d9a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6634101950 0x7f6634198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56796/0 (socket says 172.21.15.57:56796) 2023-07-20T18:58:30.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f6638d9a700 1 -- 172.21.15.57:0/3918588898 learned_addr learned my addr 172.21.15.57:0/3918588898 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:30.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f663959b700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6634100760 0x7f66341986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:30.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f6639d9c700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66341022f0 0x7f663419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:30.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f6638d9a700 1 -- 172.21.15.57:0/3918588898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6634100760 msgr2=0x7f66341986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:30.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f6638d9a700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6634100760 0x7f66341986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:30.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f6638d9a700 1 -- 172.21.15.57:0/3918588898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66341022f0 msgr2=0x7f663419fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:30.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f6638d9a700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66341022f0 0x7f663419fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:30.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f6638d9a700 1 -- 172.21.15.57:0/3918588898 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6630009c40 con 0x7f6634101950 2023-07-20T18:58:30.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.052+0000 7f663959b700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6634100760 0x7f66341986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:30.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.053+0000 7f6638d9a700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6634101950 0x7f6634198bf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f66240059d0 tx=0x7f662400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:30.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.053+0000 7f662a7fc700 1 -- 172.21.15.57:0/3918588898 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662401d070 con 0x7f6634101950 2023-07-20T18:58:30.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.053+0000 7f662a7fc700 1 -- 172.21.15.57:0/3918588898 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f662400be70 con 0x7f6634101950 2023-07-20T18:58:30.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.053+0000 7f663b7ff700 1 -- 172.21.15.57:0/3918588898 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66240095f0 con 0x7f6634101950 2023-07-20T18:58:30.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.053+0000 7f662a7fc700 1 -- 172.21.15.57:0/3918588898 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624017650 con 0x7f6634101950 2023-07-20T18:58:30.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.053+0000 7f663b7ff700 1 -- 172.21.15.57:0/3918588898 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66341a0a70 con 0x7f6634101950 2023-07-20T18:58:30.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.054+0000 7f663b7ff700 1 -- 172.21.15.57:0/3918588898 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663410c0b0 con 0x7f6634101950 2023-07-20T18:58:30.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.058+0000 7f662a7fc700 1 -- 172.21.15.57:0/3918588898 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6624022470 con 0x7f6634101950 2023-07-20T18:58:30.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.059+0000 7f662a7fc700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66200a11d0 0x7f66200a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:30.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.059+0000 7f662a7fc700 1 -- 172.21.15.57:0/3918588898 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f6624022e40 con 0x7f6634101950 2023-07-20T18:58:30.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.059+0000 7f663959b700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66200a11d0 0x7f66200a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:30.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.060+0000 7f663959b700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66200a11d0 0x7f66200a3690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f66341017b0 tx=0x7f663000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:30.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.062+0000 7f662a7fc700 1 -- 172.21.15.57:0/3918588898 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6624097ac0 con 0x7f6634101950 2023-07-20T18:58:30.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:29 smithi088 ceph-mon[128823]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:30.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:29 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/729094379' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T18:58:30.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:29 smithi088 ceph-mon[128823]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T18:58:30.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:29 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/283821544' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T18:58:30.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:29 smithi088 ceph-mon[128823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T18:58:30.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:29 smithi057 ceph-mon[126340]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:30.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:29 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/729094379' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T18:58:30.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:29 smithi057 ceph-mon[126340]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T18:58:30.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:29 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/283821544' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T18:58:30.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:29 smithi057 ceph-mon[126340]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T18:58:30.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:29 smithi192 ceph-mon[129200]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:30.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:29 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/729094379' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T18:58:30.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:29 smithi192 ceph-mon[129200]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T18:58:30.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:29 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/283821544' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T18:58:30.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:29 smithi192 ceph-mon[129200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T18:58:30.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.331+0000 7f663b7ff700 1 -- 172.21.15.57:0/3918588898 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f66341a10b0 con 0x7f6634101950 2023-07-20T18:58:30.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.839+0000 7f662a7fc700 1 -- 172.21.15.57:0/3918588898 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f662409b0e0 con 0x7f6634101950 2023-07-20T18:58:30.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.841+0000 7f663b7ff700 1 -- 172.21.15.57:0/3918588898 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66200a11d0 msgr2=0x7f66200a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:30.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.841+0000 7f663b7ff700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66200a11d0 0x7f66200a3690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f66341017b0 tx=0x7f663000b040 comp rx=0 tx=0).stop 2023-07-20T18:58:30.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.841+0000 7f663b7ff700 1 -- 172.21.15.57:0/3918588898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6634101950 msgr2=0x7f6634198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:30.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.841+0000 7f663b7ff700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6634101950 0x7f6634198bf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f66240059d0 tx=0x7f662400f690 comp rx=0 tx=0).stop 2023-07-20T18:58:30.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.842+0000 7f663b7ff700 1 -- 172.21.15.57:0/3918588898 shutdown_connections 2023-07-20T18:58:30.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.842+0000 7f663b7ff700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f66200a11d0 0x7f66200a3690 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:30.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.842+0000 7f663b7ff700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6634100760 0x7f66341986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:30.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.842+0000 7f663b7ff700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66341022f0 0x7f663419fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:30.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.842+0000 7f663b7ff700 1 --2- 172.21.15.57:0/3918588898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6634101950 0x7f6634198bf0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:30.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.842+0000 7f663b7ff700 1 -- 172.21.15.57:0/3918588898 >> 172.21.15.57:0/3918588898 conn(0x7f66340fbcb0 msgr2=0x7f66340fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:30.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.842+0000 7f663b7ff700 1 -- 172.21.15.57:0/3918588898 shutdown_connections 2023-07-20T18:58:30.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:30.842+0000 7f663b7ff700 1 -- 172.21.15.57:0/3918588898 wait complete. 2023-07-20T18:58:30.894 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 2 and data pool 3 2023-07-20T18:58:30.913 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-07-20T18:58:30.913 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-07-20T18:58:31.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.079+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2527262418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b58074750 msgr2=0x7f9b58072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:31.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.079+0000 7f9b5e9ff700 1 --2- 172.21.15.57:0/2527262418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b58074750 0x7f9b58072ba0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009990 tx=0x7f9b48009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.080+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2527262418 shutdown_connections 2023-07-20T18:58:31.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.080+0000 7f9b5e9ff700 1 --2- 172.21.15.57:0/2527262418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b58108630 0x7f9b5810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.080+0000 7f9b5e9ff700 1 --2- 172.21.15.57:0/2527262418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b58074750 0x7f9b58072ba0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.080+0000 7f9b5e9ff700 1 --2- 172.21.15.57:0/2527262418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b58073170 0x7f9b580735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.080+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2527262418 >> 172.21.15.57:0/2527262418 conn(0x7f9b580fbcb0 msgr2=0x7f9b580fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:31.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.080+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2527262418 shutdown_connections 2023-07-20T18:58:31.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.080+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2527262418 wait complete. 2023-07-20T18:58:31.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b5e9ff700 1 Processor -- start 2023-07-20T18:58:31.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b5e9ff700 1 -- start start 2023-07-20T18:58:31.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b5e9ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b58073170 0x7f9b58198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:31.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b5e9ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b58074750 0x7f9b58198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:31.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b5e9ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b58108630 0x7f9b5819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:31.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b5e9ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b581a02d0 con 0x7f9b58074750 2023-07-20T18:58:31.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b5e9ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b581a0410 con 0x7f9b58108630 2023-07-20T18:58:31.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b5e9ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b581a05a0 con 0x7f9b58073170 2023-07-20T18:58:31.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b4ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b58074750 0x7f9b58198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:31.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b4ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b58074750 0x7f9b58198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56820/0 (socket says 172.21.15.57:56820) 2023-07-20T18:58:31.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b4ffff700 1 -- 172.21.15.57:0/2818851885 learned_addr learned my addr 172.21.15.57:0/2818851885 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:31.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.081+0000 7f9b57fff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b58073170 0x7f9b58198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:31.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b4ffff700 1 -- 172.21.15.57:0/2818851885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b58073170 msgr2=0x7f9b58198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:31.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b4ffff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b58073170 0x7f9b58198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b4ffff700 1 -- 172.21.15.57:0/2818851885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b58108630 msgr2=0x7f9b5819fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:31.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b4ffff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b58108630 0x7f9b5819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b4ffff700 1 -- 172.21.15.57:0/2818851885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b480095f0 con 0x7f9b58074750 2023-07-20T18:58:31.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b4ffff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b58074750 0x7f9b58198c90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000e9a0 tx=0x7f9b4000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:31.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b55ffb700 1 -- 172.21.15.57:0/2818851885 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b40009790 con 0x7f9b58074750 2023-07-20T18:58:31.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b55ffb700 1 -- 172.21.15.57:0/2818851885 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f9b4000cb50 con 0x7f9b58074750 2023-07-20T18:58:31.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b55ffb700 1 -- 172.21.15.57:0/2818851885 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b40018680 con 0x7f9b58074750 2023-07-20T18:58:31.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2818851885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b581a0880 con 0x7f9b58074750 2023-07-20T18:58:31.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.082+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2818851885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b581a0d40 con 0x7f9b58074750 2023-07-20T18:58:31.136 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:31.136 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:30 smithi088 ceph-mon[128823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-20T18:58:31.136 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:30 smithi088 ceph-mon[128823]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T18:58:31.136 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:31.136 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:31.136 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:58:31.137 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:31.137 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:30 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:58:31.137 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:30 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3918588898' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T18:58:31.137 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:30 smithi088 ceph-mon[128823]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T18:58:31.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.083+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2818851885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b58106d30 con 0x7f9b58074750 2023-07-20T18:58:31.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.085+0000 7f9b55ffb700 1 -- 172.21.15.57:0/2818851885 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9b4000ccc0 con 0x7f9b58074750 2023-07-20T18:58:31.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.087+0000 7f9b55ffb700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9b440a11d0 0x7f9b440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:31.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.087+0000 7f9b55ffb700 1 -- 172.21.15.57:0/2818851885 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9b40014070 con 0x7f9b58074750 2023-07-20T18:58:31.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.087+0000 7f9b57fff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9b440a11d0 0x7f9b440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:31.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.088+0000 7f9b57fff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9b440a11d0 0x7f9b440a3690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800b2e0 tx=0x7f9b48005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:31.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.090+0000 7f9b55ffb700 1 -- 172.21.15.57:0/2818851885 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9b40051780 con 0x7f9b58074750 2023-07-20T18:58:31.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:31.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:30 smithi057 ceph-mon[126340]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-20T18:58:31.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:30 smithi057 ceph-mon[126340]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T18:58:31.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:31.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:31.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:58:31.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:31.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:30 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:58:31.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:30 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3918588898' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T18:58:31.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:30 smithi057 ceph-mon[126340]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T18:58:31.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:31.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:30 smithi192 ceph-mon[129200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-20T18:58:31.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:30 smithi192 ceph-mon[129200]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T18:58:31.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:31.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:31.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:58:31.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:31.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:30 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T18:58:31.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:30 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3918588898' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T18:58:31.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:30 smithi192 ceph-mon[129200]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T18:58:31.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.320+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2818851885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f9b5802c970 con 0x7f9b58074750 2023-07-20T18:58:31.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.796+0000 7f9b55ffb700 1 -- 172.21.15.57:0/2818851885 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v82) v1 ==== 157+0+0 (secure 0 0 0) 0x7f9b40096210 con 0x7f9b58074750 2023-07-20T18:58:31.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2818851885 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9b440a11d0 msgr2=0x7f9b440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:31.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9b440a11d0 0x7f9b440a3690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800b2e0 tx=0x7f9b48005fb0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2818851885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b58074750 msgr2=0x7f9b58198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:31.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b58074750 0x7f9b58198c90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000e9a0 tx=0x7f9b4000ecb0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2818851885 shutdown_connections 2023-07-20T18:58:31.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9b440a11d0 0x7f9b440a3690 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b58073170 0x7f9b58198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b58108630 0x7f9b5819fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 --2- 172.21.15.57:0/2818851885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b58074750 0x7f9b58198c90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:31.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2818851885 >> 172.21.15.57:0/2818851885 conn(0x7f9b580fbcb0 msgr2=0x7f9b580fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:31.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2818851885 shutdown_connections 2023-07-20T18:58:31.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:31.799+0000 7f9b5e9ff700 1 -- 172.21.15.57:0/2818851885 wait complete. 2023-07-20T18:58:31.869 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-07-20T18:58:32.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.042+0000 7fc0c1fff700 1 -- 172.21.15.57:0/1440796799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc100770 msgr2=0x7fc0bc100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:32.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.042+0000 7fc0c1fff700 1 --2- 172.21.15.57:0/1440796799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc100770 0x7fc0bc100b90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0099c0 tx=0x7fc0ac009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:32.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.042+0000 7fc0c1fff700 1 -- 172.21.15.57:0/1440796799 shutdown_connections 2023-07-20T18:58:32.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.042+0000 7fc0c1fff700 1 --2- 172.21.15.57:0/1440796799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0bc102300 0x7fc0bc108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:32.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.042+0000 7fc0c1fff700 1 --2- 172.21.15.57:0/1440796799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc100770 0x7fc0bc100b90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:32.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.042+0000 7fc0c1fff700 1 --2- 172.21.15.57:0/1440796799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0bc101960 0x7fc0bc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:32.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.042+0000 7fc0c1fff700 1 -- 172.21.15.57:0/1440796799 >> 172.21.15.57:0/1440796799 conn(0x7fc0bc0fbca0 msgr2=0x7fc0bc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:32.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.042+0000 7fc0c1fff700 1 -- 172.21.15.57:0/1440796799 shutdown_connections 2023-07-20T18:58:32.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.042+0000 7fc0c1fff700 1 -- 172.21.15.57:0/1440796799 wait complete. 2023-07-20T18:58:32.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.043+0000 7fc0c1fff700 1 Processor -- start 2023-07-20T18:58:32.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.044+0000 7fc0c1fff700 1 -- start start 2023-07-20T18:58:32.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.044+0000 7fc0c1fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc100770 0x7fc0bc1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:32.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.044+0000 7fc0c1fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0bc101960 0x7fc0bc196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:32.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.044+0000 7fc0c1fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0bc102300 0x7fc0bc19da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:32.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.044+0000 7fc0c1fff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc19e050 con 0x7fc0bc101960 2023-07-20T18:58:32.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.044+0000 7fc0c1fff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc06a190 con 0x7fc0bc100770 2023-07-20T18:58:32.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.044+0000 7fc0c1fff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc06a320 con 0x7fc0bc102300 2023-07-20T18:58:32.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.045+0000 7fc0bb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc100770 0x7fc0bc1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:32.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.045+0000 7fc0bb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc100770 0x7fc0bc1964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:33792/0 (socket says 172.21.15.57:33792) 2023-07-20T18:58:32.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.045+0000 7fc0bb7fe700 1 -- 172.21.15.57:0/3753674550 learned_addr learned my addr 172.21.15.57:0/3753674550 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:32.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.045+0000 7fc0baffd700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0bc101960 0x7fc0bc196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:32.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.045+0000 7fc0bb7fe700 1 -- 172.21.15.57:0/3753674550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0bc102300 msgr2=0x7fc0bc19da80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-20T18:58:32.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.045+0000 7fc0bb7fe700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0bc102300 0x7fc0bc19da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:32.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.046+0000 7fc0bb7fe700 1 -- 172.21.15.57:0/3753674550 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0bc101960 msgr2=0x7fc0bc196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:32.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.046+0000 7fc0bb7fe700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0bc101960 0x7fc0bc196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:32.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.046+0000 7fc0bb7fe700 1 -- 172.21.15.57:0/3753674550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b4009c40 con 0x7fc0bc100770 2023-07-20T18:58:32.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.046+0000 7fc0bb7fe700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc100770 0x7fc0bc1964d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0098e0 tx=0x7fc0ac00f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:32.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.046+0000 7fc0b8ff9700 1 -- 172.21.15.57:0/3753674550 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac01d070 con 0x7fc0bc100770 2023-07-20T18:58:32.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.046+0000 7fc0b8ff9700 1 -- 172.21.15.57:0/3753674550 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fc0ac005510 con 0x7fc0bc100770 2023-07-20T18:58:32.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.046+0000 7fc0b8ff9700 1 -- 172.21.15.57:0/3753674550 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac004bb0 con 0x7fc0bc100770 2023-07-20T18:58:32.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.046+0000 7fc0c1fff700 1 -- 172.21.15.57:0/3753674550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ac0095f0 con 0x7fc0bc100770 2023-07-20T18:58:32.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.046+0000 7fc0c1fff700 1 -- 172.21.15.57:0/3753674550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0bc06a7e0 con 0x7fc0bc100770 2023-07-20T18:58:32.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.047+0000 7fc0c1fff700 1 -- 172.21.15.57:0/3753674550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0bc10c110 con 0x7fc0bc100770 2023-07-20T18:58:32.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.050+0000 7fc0b8ff9700 1 -- 172.21.15.57:0/3753674550 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc0ac00fb30 con 0x7fc0bc100770 2023-07-20T18:58:32.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.051+0000 7fc0b8ff9700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc0a40a11f0 0x7fc0a40a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:32.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.051+0000 7fc0b8ff9700 1 -- 172.21.15.57:0/3753674550 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8205+0+0 (secure 0 0 0) 0x7fc0ac0cdae0 con 0x7fc0bc100770 2023-07-20T18:58:32.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.051+0000 7fc0baffd700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc0a40a11f0 0x7fc0a40a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:32.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.052+0000 7fc0baffd700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc0a40a11f0 0x7fc0a40a36b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009bf0 tx=0x7fc0b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:32.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.052+0000 7fc0b8ff9700 1 -- 172.21.15.57:0/3753674550 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc0ac097280 con 0x7fc0bc100770 2023-07-20T18:58:32.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:31 smithi088 ceph-mon[128823]: pgmap v264: 97 pgs: 14 creating+peering, 3 active+clean, 80 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:32.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:31 smithi088 ceph-mon[128823]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T18:58:32.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:31 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3918588898' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T18:58:32.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:31 smithi088 ceph-mon[128823]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T18:58:32.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:31 smithi088 ceph-mon[128823]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T18:58:32.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:31 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2818851885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-20T18:58:32.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:31 smithi057 ceph-mon[126340]: pgmap v264: 97 pgs: 14 creating+peering, 3 active+clean, 80 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:32.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:31 smithi057 ceph-mon[126340]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T18:58:32.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:31 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3918588898' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T18:58:32.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:31 smithi057 ceph-mon[126340]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T18:58:32.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:31 smithi057 ceph-mon[126340]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T18:58:32.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:31 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2818851885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-20T18:58:32.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:32.273+0000 7fc0c1fff700 1 -- 172.21.15.57:0/3753674550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc0bc10c350 con 0x7fc0bc100770 2023-07-20T18:58:32.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:31 smithi192 ceph-mon[129200]: pgmap v264: 97 pgs: 14 creating+peering, 3 active+clean, 80 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T18:58:32.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:31 smithi192 ceph-mon[129200]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T18:58:32.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:31 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3918588898' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T18:58:32.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:31 smithi192 ceph-mon[129200]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T18:58:32.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:31 smithi192 ceph-mon[129200]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T18:58:32.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:31 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2818851885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-20T18:58:33.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:32 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2818851885' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-20T18:58:33.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:32 smithi088 ceph-mon[128823]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T18:58:33.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:32 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] up:active 2023-07-20T18:58:33.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:32 smithi088 ceph-mon[128823]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:58:33.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:32 smithi088 ceph-mon[128823]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:58:33.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:32 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3753674550' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T18:58:33.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:32 smithi088 ceph-mon[128823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T18:58:33.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:32 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2818851885' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-20T18:58:33.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:32 smithi057 ceph-mon[126340]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T18:58:33.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:32 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] up:active 2023-07-20T18:58:33.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:32 smithi057 ceph-mon[126340]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:58:33.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:32 smithi057 ceph-mon[126340]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:58:33.237 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:32 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3753674550' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T18:58:33.237 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:32 smithi057 ceph-mon[126340]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T18:58:33.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:32 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2818851885' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-20T18:58:33.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:32 smithi192 ceph-mon[129200]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T18:58:33.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:32 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] up:active 2023-07-20T18:58:33.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:32 smithi192 ceph-mon[129200]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:58:33.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:32 smithi192 ceph-mon[129200]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:58:33.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:32 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3753674550' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T18:58:33.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:32 smithi192 ceph-mon[129200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T18:58:33.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.855+0000 7fc0b8ff9700 1 -- 172.21.15.57:0/3753674550 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v84) v1 ==== 228+0+0 (secure 0 0 0) 0x7fc0ac09a8a0 con 0x7fc0bc100770 2023-07-20T18:58:33.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.858+0000 7fc0c1fff700 1 -- 172.21.15.57:0/3753674550 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc0a40a11f0 msgr2=0x7fc0a40a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:33.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.858+0000 7fc0c1fff700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc0a40a11f0 0x7fc0a40a36b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009bf0 tx=0x7fc0b0009260 comp rx=0 tx=0).stop 2023-07-20T18:58:33.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.858+0000 7fc0c1fff700 1 -- 172.21.15.57:0/3753674550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc100770 msgr2=0x7fc0bc1964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:33.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.858+0000 7fc0c1fff700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc100770 0x7fc0bc1964d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0098e0 tx=0x7fc0ac00f740 comp rx=0 tx=0).stop 2023-07-20T18:58:33.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.859+0000 7fc0c1fff700 1 -- 172.21.15.57:0/3753674550 shutdown_connections 2023-07-20T18:58:33.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.859+0000 7fc0c1fff700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc0a40a11f0 0x7fc0a40a36b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:33.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.859+0000 7fc0c1fff700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0bc102300 0x7fc0bc19da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:33.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.859+0000 7fc0c1fff700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc100770 0x7fc0bc1964d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:33.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.859+0000 7fc0c1fff700 1 --2- 172.21.15.57:0/3753674550 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0bc101960 0x7fc0bc196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:33.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.859+0000 7fc0c1fff700 1 -- 172.21.15.57:0/3753674550 >> 172.21.15.57:0/3753674550 conn(0x7fc0bc0fbca0 msgr2=0x7fc0bc0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:33.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.860+0000 7fc0c1fff700 1 -- 172.21.15.57:0/3753674550 shutdown_connections 2023-07-20T18:58:33.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:33.864+0000 7fc0c1fff700 1 -- 172.21.15.57:0/3753674550 wait complete. 2023-07-20T18:58:33.918 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data_ec' created 2023-07-20T18:58:33.936 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-07-20T18:58:34.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: pgmap v267: 97 pgs: 14 creating+peering, 44 active+clean, 39 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2023-07-20T18:58:34.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: osdmap e83: 12 total, 12 up, 12 in 2023-07-20T18:58:34.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:34.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:58:34.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:33 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.096+0000 7f24c09ff700 1 -- 172.21.15.57:0/3923961945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24bc100760 msgr2=0x7f24bc100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:34.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.096+0000 7f24c09ff700 1 --2- 172.21.15.57:0/3923961945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24bc100760 0x7f24bc100b80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0099c0 tx=0x7f24ac009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:34.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.096+0000 7f24c09ff700 1 -- 172.21.15.57:0/3923961945 shutdown_connections 2023-07-20T18:58:34.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.096+0000 7f24c09ff700 1 --2- 172.21.15.57:0/3923961945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24bc1022f0 0x7f24bc108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:34.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.096+0000 7f24c09ff700 1 --2- 172.21.15.57:0/3923961945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24bc100760 0x7f24bc100b80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:34.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.096+0000 7f24c09ff700 1 --2- 172.21.15.57:0/3923961945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc101950 0x7f24bc101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:34.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.096+0000 7f24c09ff700 1 -- 172.21.15.57:0/3923961945 >> 172.21.15.57:0/3923961945 conn(0x7f24bc0fbcb0 msgr2=0x7f24bc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:34.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.096+0000 7f24c09ff700 1 -- 172.21.15.57:0/3923961945 shutdown_connections 2023-07-20T18:58:34.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.096+0000 7f24c09ff700 1 -- 172.21.15.57:0/3923961945 wait complete. 2023-07-20T18:58:34.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.097+0000 7f24c09ff700 1 Processor -- start 2023-07-20T18:58:34.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24c09ff700 1 -- start start 2023-07-20T18:58:34.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24c09ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24bc100760 0x7f24bc198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:34.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24c09ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc101950 0x7f24bc198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:34.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24c09ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24bc1022f0 0x7f24bc19fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:34.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24c09ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc1a0310 con 0x7f24bc100760 2023-07-20T18:58:34.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24c09ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc1a0450 con 0x7f24bc101950 2023-07-20T18:58:34.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24c09ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc1a05e0 con 0x7f24bc1022f0 2023-07-20T18:58:34.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24ba59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24bc100760 0x7f24bc198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:34.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24ba59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24bc100760 0x7f24bc198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56848/0 (socket says 172.21.15.57:56848) 2023-07-20T18:58:34.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24ba59c700 1 -- 172.21.15.57:0/856189758 learned_addr learned my addr 172.21.15.57:0/856189758 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:34.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24bad9d700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24bc1022f0 0x7f24bc19fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:34.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.098+0000 7f24b9d9b700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc101950 0x7f24bc198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:34.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24ba59c700 1 -- 172.21.15.57:0/856189758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24bc1022f0 msgr2=0x7f24bc19fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:34.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24ba59c700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24bc1022f0 0x7f24bc19fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:34.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24ba59c700 1 -- 172.21.15.57:0/856189758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc101950 msgr2=0x7f24bc198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:34.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24ba59c700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc101950 0x7f24bc198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:34.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24ba59c700 1 -- 172.21.15.57:0/856189758 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac0095f0 con 0x7f24bc100760 2023-07-20T18:58:34.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24b9d9b700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc101950 0x7f24bc198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T18:58:34.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24bad9d700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24bc1022f0 0x7f24bc19fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:34.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24ba59c700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24bc100760 0x7f24bc198790 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0098e0 tx=0x7f24ac00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:34.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24ab7fe700 1 -- 172.21.15.57:0/856189758 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac01d070 con 0x7f24bc100760 2023-07-20T18:58:34.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24ab7fe700 1 -- 172.21.15.57:0/856189758 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f24ac005030 con 0x7f24bc100760 2023-07-20T18:58:34.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24c09ff700 1 -- 172.21.15.57:0/856189758 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24bc1a0860 con 0x7f24bc100760 2023-07-20T18:58:34.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24ab7fe700 1 -- 172.21.15.57:0/856189758 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac004af0 con 0x7f24bc100760 2023-07-20T18:58:34.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.099+0000 7f24c09ff700 1 -- 172.21.15.57:0/856189758 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24bc1a0c80 con 0x7f24bc100760 2023-07-20T18:58:34.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.101+0000 7f24c09ff700 1 -- 172.21.15.57:0/856189758 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24bc192ef0 con 0x7f24bc100760 2023-07-20T18:58:34.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.103+0000 7f24ab7fe700 1 -- 172.21.15.57:0/856189758 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f24ac0054f0 con 0x7f24bc100760 2023-07-20T18:58:34.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.104+0000 7f24ab7fe700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f24a40a1150 0x7f24a40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:34.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.104+0000 7f24ab7fe700 1 -- 172.21.15.57:0/856189758 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f24ac0cdd80 con 0x7f24bc100760 2023-07-20T18:58:34.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.104+0000 7f24b9d9b700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f24a40a1150 0x7f24a40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:34.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.105+0000 7f24b9d9b700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f24a40a1150 0x7f24a40a3610 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009cb0 tx=0x7f24b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:34.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.107+0000 7f24ab7fe700 1 -- 172.21.15.57:0/856189758 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f24ac097330 con 0x7f24bc100760 2023-07-20T18:58:34.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: pgmap v267: 97 pgs: 14 creating+peering, 44 active+clean, 39 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2023-07-20T18:58:34.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: osdmap e83: 12 total, 12 up, 12 in 2023-07-20T18:58:34.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.237 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:34.237 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:58:34.237 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:33 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: pgmap v267: 97 pgs: 14 creating+peering, 44 active+clean, 39 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2023-07-20T18:58:34.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: osdmap e83: 12 total, 12 up, 12 in 2023-07-20T18:58:34.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:58:34.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T18:58:34.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:33 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' 2023-07-20T18:58:34.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.314+0000 7f24c09ff700 1 -- 172.21.15.57:0/856189758 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f24bc02c970 con 0x7f24bc100760 2023-07-20T18:58:34.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.861+0000 7f24ab7fe700 1 -- 172.21.15.57:0/856189758 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7f24ac09a950 con 0x7f24bc100760 2023-07-20T18:58:34.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.866+0000 7f24c09ff700 1 -- 172.21.15.57:0/856189758 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f24a40a1150 msgr2=0x7f24a40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:34.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.866+0000 7f24c09ff700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f24a40a1150 0x7f24a40a3610 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009cb0 tx=0x7f24b0009260 comp rx=0 tx=0).stop 2023-07-20T18:58:34.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.866+0000 7f24c09ff700 1 -- 172.21.15.57:0/856189758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24bc100760 msgr2=0x7f24bc198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:34.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.866+0000 7f24c09ff700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24bc100760 0x7f24bc198790 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0098e0 tx=0x7f24ac00f690 comp rx=0 tx=0).stop 2023-07-20T18:58:34.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.866+0000 7f24c09ff700 1 -- 172.21.15.57:0/856189758 shutdown_connections 2023-07-20T18:58:34.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.866+0000 7f24c09ff700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f24a40a1150 0x7f24a40a3610 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:34.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.866+0000 7f24c09ff700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24bc1022f0 0x7f24bc19fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:34.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.866+0000 7f24c09ff700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24bc100760 0x7f24bc198790 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:34.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.866+0000 7f24c09ff700 1 --2- 172.21.15.57:0/856189758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc101950 0x7f24bc198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:34.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.866+0000 7f24c09ff700 1 -- 172.21.15.57:0/856189758 >> 172.21.15.57:0/856189758 conn(0x7f24bc0fbcb0 msgr2=0x7f24bc107240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:34.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.869+0000 7f24c09ff700 1 -- 172.21.15.57:0/856189758 shutdown_connections 2023-07-20T18:58:34.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:34.870+0000 7f24c09ff700 1 -- 172.21.15.57:0/856189758 wait complete. 2023-07-20T18:58:34.930 INFO:teuthology.orchestra.run.smithi057.stderr:set pool 4 allow_ec_overwrites to true 2023-07-20T18:58:34.954 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-07-20T18:58:35.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.115+0000 7f9f435ff700 1 -- 172.21.15.57:0/3164270426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3c108620 msgr2=0x7f9f3c10aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:35.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.115+0000 7f9f435ff700 1 --2- 172.21.15.57:0/3164270426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3c108620 0x7f9f3c10aab0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9f3400b0a0 tx=0x7f9f3400b3b0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.115+0000 7f9f435ff700 1 -- 172.21.15.57:0/3164270426 shutdown_connections 2023-07-20T18:58:35.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.115+0000 7f9f435ff700 1 --2- 172.21.15.57:0/3164270426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f3c0747b0 0x7f9f3c072c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.115+0000 7f9f435ff700 1 --2- 172.21.15.57:0/3164270426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3c108620 0x7f9f3c10aab0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.115+0000 7f9f435ff700 1 --2- 172.21.15.57:0/3164270426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c0731d0 0x7f9f3c073650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.115+0000 7f9f435ff700 1 -- 172.21.15.57:0/3164270426 >> 172.21.15.57:0/3164270426 conn(0x7f9f3c0fbcb0 msgr2=0x7f9f3c0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.115+0000 7f9f435ff700 1 -- 172.21.15.57:0/3164270426 shutdown_connections 2023-07-20T18:58:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.115+0000 7f9f435ff700 1 -- 172.21.15.57:0/3164270426 wait complete. 2023-07-20T18:58:35.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.116+0000 7f9f435ff700 1 Processor -- start 2023-07-20T18:58:35.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.116+0000 7f9f435ff700 1 -- start start 2023-07-20T18:58:35.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.116+0000 7f9f435ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f3c0731d0 0x7f9f3c198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:35.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.116+0000 7f9f435ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c0747b0 0x7f9f3c198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:35.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.116+0000 7f9f435ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3c108620 0x7f9f3c19fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:35.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f435ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c1a03e0 con 0x7f9f3c108620 2023-07-20T18:58:35.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f435ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c1a0520 con 0x7f9f3c0747b0 2023-07-20T18:58:35.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f435ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c1a06b0 con 0x7f9f3c0731d0 2023-07-20T18:58:35.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f4139b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f3c0731d0 0x7f9f3c198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:35.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f4139b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f3c0731d0 0x7f9f3c198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:60802/0 (socket says 172.21.15.57:60802) 2023-07-20T18:58:35.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f4139b700 1 -- 172.21.15.57:0/3340791297 learned_addr learned my addr 172.21.15.57:0/3340791297 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:35.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f41b9c700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3c108620 0x7f9f3c19fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:35.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f40b9a700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c0747b0 0x7f9f3c198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:35.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f41b9c700 1 -- 172.21.15.57:0/3340791297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f3c0731d0 msgr2=0x7f9f3c198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:35.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f41b9c700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f3c0731d0 0x7f9f3c198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f41b9c700 1 -- 172.21.15.57:0/3340791297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c0747b0 msgr2=0x7f9f3c198da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:35.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f41b9c700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c0747b0 0x7f9f3c198da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f41b9c700 1 -- 172.21.15.57:0/3340791297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f34009d00 con 0x7f9f3c108620 2023-07-20T18:58:35.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f4139b700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f3c0731d0 0x7f9f3c198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:35.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.117+0000 7f9f41b9c700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3c108620 0x7f9f3c19fe10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009fd0 tx=0x7f9f3400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:35.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.118+0000 7f9f327fc700 1 -- 172.21.15.57:0/3340791297 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3400e040 con 0x7f9f3c108620 2023-07-20T18:58:35.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.118+0000 7f9f327fc700 1 -- 172.21.15.57:0/3340791297 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f9f34004e60 con 0x7f9f3c108620 2023-07-20T18:58:35.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.118+0000 7f9f435ff700 1 -- 172.21.15.57:0/3340791297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3c1a0930 con 0x7f9f3c108620 2023-07-20T18:58:35.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.118+0000 7f9f327fc700 1 -- 172.21.15.57:0/3340791297 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f340052a0 con 0x7f9f3c108620 2023-07-20T18:58:35.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.118+0000 7f9f435ff700 1 -- 172.21.15.57:0/3340791297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3c1a0e00 con 0x7f9f3c108620 2023-07-20T18:58:35.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.119+0000 7f9f435ff700 1 -- 172.21.15.57:0/3340791297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3c106cd0 con 0x7f9f3c108620 2023-07-20T18:58:35.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.121+0000 7f9f327fc700 1 -- 172.21.15.57:0/3340791297 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9f34005400 con 0x7f9f3c108620 2023-07-20T18:58:35.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.123+0000 7f9f327fc700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9f280a11d0 0x7f9f280a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:35.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.123+0000 7f9f327fc700 1 -- 172.21.15.57:0/3340791297 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f9f340ce060 con 0x7f9f3c108620 2023-07-20T18:58:35.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.123+0000 7f9f4139b700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9f280a11d0 0x7f9f280a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:35.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.124+0000 7f9f4139b700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9f280a11d0 0x7f9f280a3690 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9f380104e0 tx=0x7f9f38007480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:35.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.127+0000 7f9f327fc700 1 -- 172.21.15.57:0/3340791297 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9f34100780 con 0x7f9f3c108620 2023-07-20T18:58:35.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:34 smithi057 ceph-mon[126340]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T18:58:35.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:34 smithi057 ceph-mon[126340]: osdmap e84: 12 total, 12 up, 12 in 2023-07-20T18:58:35.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:34 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/856189758' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-20T18:58:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:34 smithi192 ceph-mon[129200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T18:58:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:34 smithi192 ceph-mon[129200]: osdmap e84: 12 total, 12 up, 12 in 2023-07-20T18:58:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:34 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/856189758' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-20T18:58:35.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:34 smithi088 ceph-mon[128823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T18:58:35.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:34 smithi088 ceph-mon[128823]: osdmap e84: 12 total, 12 up, 12 in 2023-07-20T18:58:35.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:34 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/856189758' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-20T18:58:35.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.382+0000 7f9f435ff700 1 -- 172.21.15.57:0/3340791297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f9f3c02c970 con 0x7f9f3c108620 2023-07-20T18:58:35.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.863+0000 7f9f327fc700 1 -- 172.21.15.57:0/3340791297 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f9f34097610 con 0x7f9f3c108620 2023-07-20T18:58:35.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.867+0000 7f9f435ff700 1 -- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9f280a11d0 msgr2=0x7f9f280a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:35.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.867+0000 7f9f435ff700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9f280a11d0 0x7f9f280a3690 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9f380104e0 tx=0x7f9f38007480 comp rx=0 tx=0).stop 2023-07-20T18:58:35.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.867+0000 7f9f435ff700 1 -- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3c108620 msgr2=0x7f9f3c19fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:35.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.867+0000 7f9f435ff700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3c108620 0x7f9f3c19fe10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009fd0 tx=0x7f9f3400bce0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.867+0000 7f9f435ff700 1 -- 172.21.15.57:0/3340791297 shutdown_connections 2023-07-20T18:58:35.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.867+0000 7f9f435ff700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f9f280a11d0 0x7f9f280a3690 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.867+0000 7f9f435ff700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f3c0731d0 0x7f9f3c198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.867+0000 7f9f435ff700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3c108620 0x7f9f3c19fe10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.867+0000 7f9f435ff700 1 --2- 172.21.15.57:0/3340791297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c0747b0 0x7f9f3c198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:35.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.868+0000 7f9f435ff700 1 -- 172.21.15.57:0/3340791297 >> 172.21.15.57:0/3340791297 conn(0x7f9f3c0fbcb0 msgr2=0x7f9f3c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:35.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.868+0000 7f9f435ff700 1 -- 172.21.15.57:0/3340791297 shutdown_connections 2023-07-20T18:58:35.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:35.868+0000 7f9f435ff700 1 -- 172.21.15.57:0/3340791297 wait complete. 2023-07-20T18:58:35.919 INFO:teuthology.orchestra.run.smithi057.stderr:added data pool 4 to fsmap 2023-07-20T18:58:35.939 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:36.132 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:35 smithi088 ceph-mon[128823]: pgmap v270: 129 pgs: 12 creating+peering, 89 active+clean, 28 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-07-20T18:58:36.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:35 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/856189758' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-20T18:58:36.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:35 smithi088 ceph-mon[128823]: osdmap e85: 12 total, 12 up, 12 in 2023-07-20T18:58:36.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:35 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3340791297' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-20T18:58:36.133 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 18:58:35 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-b[154175]: 2023-07-20T18:58:35.771+0000 7fc19c7e5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-20T18:58:36.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.098+0000 7f35099ff700 1 -- 172.21.15.57:0/3157505415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3504102300 msgr2=0x7f3504108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:36.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.098+0000 7f35099ff700 1 --2- 172.21.15.57:0/3157505415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3504102300 0x7f3504108bb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f34fc00b0d0 tx=0x7f34fc00b3e0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.098+0000 7f35099ff700 1 -- 172.21.15.57:0/3157505415 shutdown_connections 2023-07-20T18:58:36.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.098+0000 7f35099ff700 1 --2- 172.21.15.57:0/3157505415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3504100770 0x7f3504100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.098+0000 7f35099ff700 1 --2- 172.21.15.57:0/3157505415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3504102300 0x7f3504108bb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.098+0000 7f35099ff700 1 --2- 172.21.15.57:0/3157505415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3504101960 0x7f3504101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.098+0000 7f35099ff700 1 -- 172.21.15.57:0/3157505415 >> 172.21.15.57:0/3157505415 conn(0x7f35040fbca0 msgr2=0x7f35040fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:36.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.098+0000 7f35099ff700 1 -- 172.21.15.57:0/3157505415 shutdown_connections 2023-07-20T18:58:36.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.098+0000 7f35099ff700 1 -- 172.21.15.57:0/3157505415 wait complete. 2023-07-20T18:58:36.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.099+0000 7f35099ff700 1 Processor -- start 2023-07-20T18:58:36.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.099+0000 7f35099ff700 1 -- start start 2023-07-20T18:58:36.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35099ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3504100770 0x7f35041987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:36.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35099ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3504101960 0x7f3504198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:36.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35099ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3504102300 0x7f350419fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:36.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35099ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35041a0330 con 0x7f3504102300 2023-07-20T18:58:36.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35099ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35041a0470 con 0x7f3504101960 2023-07-20T18:58:36.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35099ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35041a0600 con 0x7f3504100770 2023-07-20T18:58:36.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35037fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3504102300 0x7f350419fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:36.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35037fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3504102300 0x7f350419fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44716/0 (socket says 172.21.15.57:44716) 2023-07-20T18:58:36.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35037fe700 1 -- 172.21.15.57:0/2886114833 learned_addr learned my addr 172.21.15.57:0/2886114833 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:36.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f3502ffd700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3504100770 0x7f35041987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:36.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35027fc700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3504101960 0x7f3504198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:36.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35037fe700 1 -- 172.21.15.57:0/2886114833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3504100770 msgr2=0x7f35041987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:36.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35037fe700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3504100770 0x7f35041987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35037fe700 1 -- 172.21.15.57:0/2886114833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3504101960 msgr2=0x7f3504198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:36.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35037fe700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3504101960 0x7f3504198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35037fe700 1 -- 172.21.15.57:0/2886114833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc009d00 con 0x7f3504102300 2023-07-20T18:58:36.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.100+0000 7f35027fc700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3504101960 0x7f3504198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T18:58:36.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.101+0000 7f35037fe700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3504102300 0x7f350419fd60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f34fc00d010 tx=0x7f34fc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:36.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.101+0000 7f35089fd700 1 -- 172.21.15.57:0/2886114833 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc00e040 con 0x7f3504102300 2023-07-20T18:58:36.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.101+0000 7f35089fd700 1 -- 172.21.15.57:0/2886114833 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f34fc01dd70 con 0x7f3504102300 2023-07-20T18:58:36.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.101+0000 7f35099ff700 1 -- 172.21.15.57:0/2886114833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35041a0880 con 0x7f3504102300 2023-07-20T18:58:36.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.101+0000 7f35089fd700 1 -- 172.21.15.57:0/2886114833 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc012e80 con 0x7f3504102300 2023-07-20T18:58:36.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.101+0000 7f35099ff700 1 -- 172.21.15.57:0/2886114833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35041a0d50 con 0x7f3504102300 2023-07-20T18:58:36.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.102+0000 7f35099ff700 1 -- 172.21.15.57:0/2886114833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350410c0d0 con 0x7f3504102300 2023-07-20T18:58:36.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.104+0000 7f35089fd700 1 -- 172.21.15.57:0/2886114833 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f34fc012470 con 0x7f3504102300 2023-07-20T18:58:36.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.105+0000 7f35089fd700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f34ec0a1220 0x7f34ec0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:36.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.106+0000 7f35089fd700 1 -- 172.21.15.57:0/2886114833 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f34fc0ce3f0 con 0x7f3504102300 2023-07-20T18:58:36.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.106+0000 7f3502ffd700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f34ec0a1220 0x7f34ec0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:36.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.106+0000 7f3502ffd700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f34ec0a1220 0x7f34ec0a36e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f35041017c0 tx=0x7f34f4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:36.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.109+0000 7f35089fd700 1 -- 172.21.15.57:0/2886114833 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f34fc097980 con 0x7f3504102300 2023-07-20T18:58:36.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:35 smithi057 ceph-mon[126340]: pgmap v270: 129 pgs: 12 creating+peering, 89 active+clean, 28 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-07-20T18:58:36.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:35 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/856189758' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-20T18:58:36.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:35 smithi057 ceph-mon[126340]: osdmap e85: 12 total, 12 up, 12 in 2023-07-20T18:58:36.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:35 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3340791297' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-20T18:58:36.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:35 smithi192 ceph-mon[129200]: pgmap v270: 129 pgs: 12 creating+peering, 89 active+clean, 28 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-07-20T18:58:36.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:35 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/856189758' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-20T18:58:36.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:35 smithi192 ceph-mon[129200]: osdmap e85: 12 total, 12 up, 12 in 2023-07-20T18:58:36.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:35 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3340791297' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-20T18:58:36.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.370+0000 7f35099ff700 1 -- 172.21.15.57:0/2886114833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f350410c2f0 con 0x7f3504102300 2023-07-20T18:58:36.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.371+0000 7f35089fd700 1 -- 172.21.15.57:0/2886114833 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10763 (secure 0 0 0) 0x7f34fc09afa0 con 0x7f3504102300 2023-07-20T18:58:36.420 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:36.420 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/3251478878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":3251478878},{"type":"v1","addr":"172.21.15.57:6837","nonce":3251478878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14502,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1597260444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1597260444},{"type":"v1","addr":"172.21.15.57:6839","nonce":1597260444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/1188419010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":1188419010},{"type":"v1","addr":"172.21.15.57:6841","nonce":1188419010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2219652585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2219652585},{"type":"v1","addr":"172.21.15.192:6835","nonce":2219652585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1373743783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1373743783},{"type":"v1","addr":"172.21.15.88:6835","nonce":1373743783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2391717255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2391717255},{"type":"v1","addr":"172.21.15.192:6837","nonce":2391717255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/678288704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":678288704},{"type":"v1","addr":"172.21.15.88:6837","nonce":678288704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/1456232015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1456232015},{"type":"v1","addr":"172.21.15.192:6839","nonce":1456232015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-20T18:58:30.332763+0000","modified":"2023-07-20T18:58:35.858060+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24566},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/2098157844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2098157844},{"type":"v1","addr":"172.21.15.88:6839","nonce":2098157844}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-20T18:58:36.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.373+0000 7f35099ff700 1 -- 172.21.15.57:0/2886114833 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f34ec0a1220 msgr2=0x7f34ec0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:36.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.373+0000 7f35099ff700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f34ec0a1220 0x7f34ec0a36e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f35041017c0 tx=0x7f34f4009260 comp rx=0 tx=0).stop 2023-07-20T18:58:36.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.373+0000 7f35099ff700 1 -- 172.21.15.57:0/2886114833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3504102300 msgr2=0x7f350419fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:36.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.373+0000 7f35099ff700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3504102300 0x7f350419fd60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f34fc00d010 tx=0x7f34fc00bd50 comp rx=0 tx=0).stop 2023-07-20T18:58:36.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.373+0000 7f35099ff700 1 -- 172.21.15.57:0/2886114833 shutdown_connections 2023-07-20T18:58:36.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.373+0000 7f35099ff700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f34ec0a1220 0x7f34ec0a36e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.373+0000 7f35099ff700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3504100770 0x7f35041987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.373+0000 7f35099ff700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3504102300 0x7f350419fd60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.373+0000 7f35099ff700 1 --2- 172.21.15.57:0/2886114833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3504101960 0x7f3504198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.373+0000 7f35099ff700 1 -- 172.21.15.57:0/2886114833 >> 172.21.15.57:0/2886114833 conn(0x7f35040fbca0 msgr2=0x7f3504107150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:36.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.374+0000 7f35099ff700 1 -- 172.21.15.57:0/2886114833 shutdown_connections 2023-07-20T18:58:36.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.374+0000 7f35099ff700 1 -- 172.21.15.57:0/2886114833 wait complete. 2023-07-20T18:58:36.425 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 17 2023-07-20T18:58:36.442 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T18:58:36.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.598+0000 7f31407ff700 1 -- 172.21.15.57:0/2238437691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3138101960 msgr2=0x7f3138101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:36.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.598+0000 7f31407ff700 1 --2- 172.21.15.57:0/2238437691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3138101960 0x7f3138101dc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f31280099c0 tx=0x7f3128009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.598+0000 7f31407ff700 1 -- 172.21.15.57:0/2238437691 shutdown_connections 2023-07-20T18:58:36.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.598+0000 7f31407ff700 1 --2- 172.21.15.57:0/2238437691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3138102300 0x7f3138108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.598+0000 7f31407ff700 1 --2- 172.21.15.57:0/2238437691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3138100770 0x7f3138100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.598+0000 7f31407ff700 1 --2- 172.21.15.57:0/2238437691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3138101960 0x7f3138101dc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.598+0000 7f31407ff700 1 -- 172.21.15.57:0/2238437691 >> 172.21.15.57:0/2238437691 conn(0x7f31380fbca0 msgr2=0x7f31380fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:36.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.598+0000 7f31407ff700 1 -- 172.21.15.57:0/2238437691 shutdown_connections 2023-07-20T18:58:36.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.598+0000 7f31407ff700 1 -- 172.21.15.57:0/2238437691 wait complete. 2023-07-20T18:58:36.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.599+0000 7f31407ff700 1 Processor -- start 2023-07-20T18:58:36.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.599+0000 7f31407ff700 1 -- start start 2023-07-20T18:58:36.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f31407ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3138100770 0x7f3138198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:36.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f31407ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3138101960 0x7f3138198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:36.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f31407ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3138102300 0x7f313819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:36.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f31407ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31381a02f0 con 0x7f3138102300 2023-07-20T18:58:36.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f31407ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31381a0430 con 0x7f3138101960 2023-07-20T18:58:36.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f31407ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31381a05c0 con 0x7f3138100770 2023-07-20T18:58:36.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f313ed9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3138102300 0x7f313819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:36.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f313ed9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3138102300 0x7f313819fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44720/0 (socket says 172.21.15.57:44720) 2023-07-20T18:58:36.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f313ed9c700 1 -- 172.21.15.57:0/832109962 learned_addr learned my addr 172.21.15.57:0/832109962 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:36.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f313e59b700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3138100770 0x7f3138198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:36.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f313dd9a700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3138101960 0x7f3138198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:36.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f313ed9c700 1 -- 172.21.15.57:0/832109962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3138100770 msgr2=0x7f3138198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:36.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f313ed9c700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3138100770 0x7f3138198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f313ed9c700 1 -- 172.21.15.57:0/832109962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3138101960 msgr2=0x7f3138198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:36.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f313ed9c700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3138101960 0x7f3138198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.600+0000 7f313ed9c700 1 -- 172.21.15.57:0/832109962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31280095f0 con 0x7f3138102300 2023-07-20T18:58:36.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.601+0000 7f313ed9c700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3138102300 0x7f313819fd20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f313000ef30 tx=0x7f313000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:36.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.601+0000 7f312f7fe700 1 -- 172.21.15.57:0/832109962 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3130018070 con 0x7f3138102300 2023-07-20T18:58:36.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.601+0000 7f31407ff700 1 -- 172.21.15.57:0/832109962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31381a0840 con 0x7f3138102300 2023-07-20T18:58:36.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.601+0000 7f312f7fe700 1 -- 172.21.15.57:0/832109962 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f313000f040 con 0x7f3138102300 2023-07-20T18:58:36.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.601+0000 7f312f7fe700 1 -- 172.21.15.57:0/832109962 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313000cd50 con 0x7f3138102300 2023-07-20T18:58:36.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.601+0000 7f31407ff700 1 -- 172.21.15.57:0/832109962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31381a0d00 con 0x7f3138102300 2023-07-20T18:58:36.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.602+0000 7f31407ff700 1 -- 172.21.15.57:0/832109962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313810c110 con 0x7f3138102300 2023-07-20T18:58:36.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.604+0000 7f312f7fe700 1 -- 172.21.15.57:0/832109962 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3130003a00 con 0x7f3138102300 2023-07-20T18:58:36.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.606+0000 7f312f7fe700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f31240a11a0 0x7f31240a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:36.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.607+0000 7f312f7fe700 1 -- 172.21.15.57:0/832109962 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f31300cc630 con 0x7f3138102300 2023-07-20T18:58:36.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.607+0000 7f313e59b700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f31240a11a0 0x7f31240a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:36.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.607+0000 7f313e59b700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f31240a11a0 0x7f31240a3660 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f31381017c0 tx=0x7f3134009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:36.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.610+0000 7f312f7fe700 1 -- 172.21.15.57:0/832109962 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3130095bc0 con 0x7f3138102300 2023-07-20T18:58:36.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.821+0000 7f31407ff700 1 -- 172.21.15.57:0/832109962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31381a1340 con 0x7f3138102300 2023-07-20T18:58:36.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.823+0000 7f312f7fe700 1 -- 172.21.15.57:0/832109962 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23911 (secure 0 0 0) 0x7f31300991e0 con 0x7f3138102300 2023-07-20T18:58:36.872 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:36.872 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":86,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","created":"2023-07-20T18:51:01.245641+0000","modified":"2023-07-20T18:58:35.383850+0000","last_up_change":"2023-07-20T18:57:44.823544+0000","last_in_change":"2023-07-20T18:57:27.428782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T18:54:23.215777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T18:58:28.447992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T18:58:29.772740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T18:58:33.395093+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"627a67ea-1366-4b3f-babd-067f75bad1e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6803","nonce":3023441607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6805","nonce":3023441607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6809","nonce":3023441607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6807","nonce":3023441607}]},"public_addr":"172.21.15.57:6803/3023441607","cluster_addr":"172.21.15.57:6805/3023441607","heartbeat_back_addr":"172.21.15.57:6809/3023441607","heartbeat_front_addr":"172.21.15.57:6807/3023441607","state":["exists","up"]},{"osd":1,"uuid":"adcdb236-334e-4478-98a8-7d0628492082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6801","nonce":1809528555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6803","nonce":1809528555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6807","nonce":1809528555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6805","nonce":1809528555}]},"public_addr":"172.21.15.88:6801/1809528555","cluster_addr":"172.21.15.88:6803/1809528555","heartbeat_back_addr":"172.21.15.88:6807/1809528555","heartbeat_front_addr":"172.21.15.88:6805/1809528555","state":["exists","up"]},{"osd":2,"uuid":"3cd56229-2067-433a-b9f9-f47b1b878b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6801","nonce":3891687382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6803","nonce":3891687382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6807","nonce":3891687382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6805","nonce":3891687382}]},"public_addr":"172.21.15.192:6801/3891687382","cluster_addr":"172.21.15.192:6803/3891687382","heartbeat_back_addr":"172.21.15.192:6807/3891687382","heartbeat_front_addr":"172.21.15.192:6805/3891687382","state":["exists","up"]},{"osd":3,"uuid":"aae250ab-e242-4ef8-9c3a-a4fa36102c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6811","nonce":2920717314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6813","nonce":2920717314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6817","nonce":2920717314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6815","nonce":2920717314}]},"public_addr":"172.21.15.57:6811/2920717314","cluster_addr":"172.21.15.57:6813/2920717314","heartbeat_back_addr":"172.21.15.57:6817/2920717314","heartbeat_front_addr":"172.21.15.57:6815/2920717314","state":["exists","up"]},{"osd":4,"uuid":"8c8d1dfa-1cff-4f0a-ae72-22bd522ed696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6809","nonce":3911525610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6811","nonce":3911525610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6815","nonce":3911525610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6813","nonce":3911525610}]},"public_addr":"172.21.15.88:6809/3911525610","cluster_addr":"172.21.15.88:6811/3911525610","heartbeat_back_addr":"172.21.15.88:6815/3911525610","heartbeat_front_addr":"172.21.15.88:6813/3911525610","state":["exists","up"]},{"osd":5,"uuid":"5a2a56ab-2af9-4176-ba4d-b2138f0e7c75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6809","nonce":905436397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6811","nonce":905436397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6815","nonce":905436397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6813","nonce":905436397}]},"public_addr":"172.21.15.192:6809/905436397","cluster_addr":"172.21.15.192:6811/905436397","heartbeat_back_addr":"172.21.15.192:6815/905436397","heartbeat_front_addr":"172.21.15.192:6813/905436397","state":["exists","up"]},{"osd":6,"uuid":"03eafdda-278c-443c-b376-677eeb50262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6819","nonce":333894423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6821","nonce":333894423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6825","nonce":333894423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6823","nonce":333894423}]},"public_addr":"172.21.15.57:6819/333894423","cluster_addr":"172.21.15.57:6821/333894423","heartbeat_back_addr":"172.21.15.57:6825/333894423","heartbeat_front_addr":"172.21.15.57:6823/333894423","state":["exists","up"]},{"osd":7,"uuid":"3f01254e-be73-41c8-b979-3b6a7534e2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6817","nonce":3031485551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6819","nonce":3031485551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6823","nonce":3031485551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6821","nonce":3031485551}]},"public_addr":"172.21.15.88:6817/3031485551","cluster_addr":"172.21.15.88:6819/3031485551","heartbeat_back_addr":"172.21.15.88:6823/3031485551","heartbeat_front_addr":"172.21.15.88:6821/3031485551","state":["exists","up"]},{"osd":8,"uuid":"624a150e-f47b-4a76-8785-c818dd65342e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6817","nonce":918056390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6819","nonce":918056390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6823","nonce":918056390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6821","nonce":918056390}]},"public_addr":"172.21.15.192:6817/918056390","cluster_addr":"172.21.15.192:6819/918056390","heartbeat_back_addr":"172.21.15.192:6823/918056390","heartbeat_front_addr":"172.21.15.192:6821/918056390","state":["exists","up"]},{"osd":9,"uuid":"6d9cb69d-ed7b-4009-9ec5-027becc577d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6827","nonce":1773340763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6829","nonce":1773340763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6833","nonce":1773340763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6831","nonce":1773340763}]},"public_addr":"172.21.15.57:6827/1773340763","cluster_addr":"172.21.15.57:6829/1773340763","heartbeat_back_addr":"172.21.15.57:6833/1773340763","heartbeat_front_addr":"172.21.15.57:6831/1773340763","state":["exists","up"]},{"osd":10,"uuid":"d5799219-0979-40b1-a596-acd37f37a72d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6825","nonce":2781975579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6827","nonce":2781975579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6831","nonce":2781975579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6829","nonce":2781975579}]},"public_addr":"172.21.15.88:6825/2781975579","cluster_addr":"172.21.15.88:6827/2781975579","heartbeat_back_addr":"172.21.15.88:6831/2781975579","heartbeat_front_addr":"172.21.15.88:6829/2781975579","state":["exists","up"]},{"osd":11,"uuid":"e77cac6c-107e-41f9-8efd-e1140c4e3bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6825","nonce":2864190031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6827","nonce":2864190031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6831","nonce":2864190031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6829","nonce":2864190031}]},"public_addr":"172.21.15.192:6825/2864190031","cluster_addr":"172.21.15.192:6827/2864190031","heartbeat_back_addr":"172.21.15.192:6831/2864190031","heartbeat_front_addr":"172.21.15.192:6829/2864190031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:53:38.180971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:00.253039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:20.776122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:48.631094+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:09.288231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:29.595802+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:55.089126+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:15.637849+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:35.985101+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:01.544767+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:22.351743+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:43.255157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6801/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/3804015220":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6800/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1582727629":"2023-07-21T18:51:34.614904+0000","172.21.15.57:6800/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/2044626900":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/748511752":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1151100650":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6801/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/3446341201":"2023-07-21T18:52:02.610729+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T18:58:36.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 -- 172.21.15.57:0/832109962 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f31240a11a0 msgr2=0x7f31240a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:36.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f31240a11a0 0x7f31240a3660 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f31381017c0 tx=0x7f3134009260 comp rx=0 tx=0).stop 2023-07-20T18:58:36.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 -- 172.21.15.57:0/832109962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3138102300 msgr2=0x7f313819fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:36.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3138102300 0x7f313819fd20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f313000ef30 tx=0x7f313000c3a0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 -- 172.21.15.57:0/832109962 shutdown_connections 2023-07-20T18:58:36.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f31240a11a0 0x7f31240a3660 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3138100770 0x7f3138198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3138102300 0x7f313819fd20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 --2- 172.21.15.57:0/832109962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3138101960 0x7f3138198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:36.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 -- 172.21.15.57:0/832109962 >> 172.21.15.57:0/832109962 conn(0x7f31380fbca0 msgr2=0x7f31381071d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:36.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.825+0000 7f31407ff700 1 -- 172.21.15.57:0/832109962 shutdown_connections 2023-07-20T18:58:36.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:36.826+0000 7f31407ff700 1 -- 172.21.15.57:0/832109962 wait complete. 2023-07-20T18:58:36.894 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T18:58:37.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.050+0000 7f40701ff700 1 -- 172.21.15.57:0/3729085599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4068100780 msgr2=0x7f4068100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:37.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.050+0000 7f40701ff700 1 --2- 172.21.15.57:0/3729085599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4068100780 0x7f4068100ba0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f40640099c0 tx=0x7f4064009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.050+0000 7f40701ff700 1 -- 172.21.15.57:0/3729085599 shutdown_connections 2023-07-20T18:58:37.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.050+0000 7f40701ff700 1 --2- 172.21.15.57:0/3729085599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4068102310 0x7f4068108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.050+0000 7f40701ff700 1 --2- 172.21.15.57:0/3729085599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4068100780 0x7f4068100ba0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.050+0000 7f40701ff700 1 --2- 172.21.15.57:0/3729085599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4068101970 0x7f4068101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.051+0000 7f40701ff700 1 -- 172.21.15.57:0/3729085599 >> 172.21.15.57:0/3729085599 conn(0x7f40680fbcb0 msgr2=0x7f40680fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:37.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.051+0000 7f40701ff700 1 -- 172.21.15.57:0/3729085599 shutdown_connections 2023-07-20T18:58:37.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.051+0000 7f40701ff700 1 -- 172.21.15.57:0/3729085599 wait complete. 2023-07-20T18:58:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f40701ff700 1 Processor -- start 2023-07-20T18:58:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f40701ff700 1 -- start start 2023-07-20T18:58:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f40701ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4068100780 0x7f40681986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f40701ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4068101970 0x7f4068198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f40701ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4068102310 0x7f406819fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f40701ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40681a0240 con 0x7f4068100780 2023-07-20T18:58:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f40701ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40681a0380 con 0x7f4068101970 2023-07-20T18:58:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f40701ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40681a0510 con 0x7f4068102310 2023-07-20T18:58:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f406df9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4068100780 0x7f40681986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f406df9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4068100780 0x7f40681986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44728/0 (socket says 172.21.15.57:44728) 2023-07-20T18:58:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.052+0000 7f406df9b700 1 -- 172.21.15.57:0/2704814546 learned_addr learned my addr 172.21.15.57:0/2704814546 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f406e79c700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4068102310 0x7f406819fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f406d79a700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4068101970 0x7f4068198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:37.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f406df9b700 1 -- 172.21.15.57:0/2704814546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4068102310 msgr2=0x7f406819fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:37.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f406df9b700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4068102310 0x7f406819fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f406df9b700 1 -- 172.21.15.57:0/2704814546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4068101970 msgr2=0x7f4068198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:37.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f406df9b700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4068101970 0x7f4068198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f406df9b700 1 -- 172.21.15.57:0/2704814546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40640095f0 con 0x7f4068100780 2023-07-20T18:58:37.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f406df9b700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4068100780 0x7f40681986c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4064006010 tx=0x7f4064004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:37.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f405effd700 1 -- 172.21.15.57:0/2704814546 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406401d070 con 0x7f4068100780 2023-07-20T18:58:37.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f405effd700 1 -- 172.21.15.57:0/2704814546 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f40640040a0 con 0x7f4068100780 2023-07-20T18:58:37.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f405effd700 1 -- 172.21.15.57:0/2704814546 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064005290 con 0x7f4068100780 2023-07-20T18:58:37.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f40701ff700 1 -- 172.21.15.57:0/2704814546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40681a0790 con 0x7f4068100780 2023-07-20T18:58:37.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.053+0000 7f40701ff700 1 -- 172.21.15.57:0/2704814546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40681a0b60 con 0x7f4068100780 2023-07-20T18:58:37.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.054+0000 7f40701ff700 1 -- 172.21.15.57:0/2704814546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406810c120 con 0x7f4068100780 2023-07-20T18:58:37.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.056+0000 7f405effd700 1 -- 172.21.15.57:0/2704814546 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f40640048e0 con 0x7f4068100780 2023-07-20T18:58:37.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.058+0000 7f405effd700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40540a11a0 0x7f40540a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:37.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.058+0000 7f405effd700 1 -- 172.21.15.57:0/2704814546 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f40640cd9e0 con 0x7f4068100780 2023-07-20T18:58:37.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.058+0000 7f406d79a700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40540a11a0 0x7f40540a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:37.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.058+0000 7f406d79a700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40540a11a0 0x7f40540a3660 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4058007c60 tx=0x7f40580073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:37.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.061+0000 7f405effd700 1 -- 172.21.15.57:0/2704814546 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4064096f70 con 0x7f4068100780 2023-07-20T18:58:37.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:36 smithi057 ceph-mon[126340]: osdmap e86: 12 total, 12 up, 12 in 2023-07-20T18:58:37.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:36 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3340791297' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-20T18:58:37.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:36 smithi057 ceph-mon[126340]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:58:37.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:36 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2886114833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:37.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:36 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/832109962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:37.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.271+0000 7f40701ff700 1 -- 172.21.15.57:0/2704814546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f406810c310 con 0x7f4068100780 2023-07-20T18:58:37.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.273+0000 7f405effd700 1 -- 172.21.15.57:0/2704814546 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23911 (secure 0 0 0) 0x7f406409a590 con 0x7f4068100780 2023-07-20T18:58:37.322 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:37.323 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":86,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","created":"2023-07-20T18:51:01.245641+0000","modified":"2023-07-20T18:58:35.383850+0000","last_up_change":"2023-07-20T18:57:44.823544+0000","last_in_change":"2023-07-20T18:57:27.428782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T18:54:23.215777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T18:58:28.447992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T18:58:29.772740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T18:58:33.395093+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"627a67ea-1366-4b3f-babd-067f75bad1e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6803","nonce":3023441607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6805","nonce":3023441607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6809","nonce":3023441607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6807","nonce":3023441607}]},"public_addr":"172.21.15.57:6803/3023441607","cluster_addr":"172.21.15.57:6805/3023441607","heartbeat_back_addr":"172.21.15.57:6809/3023441607","heartbeat_front_addr":"172.21.15.57:6807/3023441607","state":["exists","up"]},{"osd":1,"uuid":"adcdb236-334e-4478-98a8-7d0628492082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6801","nonce":1809528555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6803","nonce":1809528555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6807","nonce":1809528555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6805","nonce":1809528555}]},"public_addr":"172.21.15.88:6801/1809528555","cluster_addr":"172.21.15.88:6803/1809528555","heartbeat_back_addr":"172.21.15.88:6807/1809528555","heartbeat_front_addr":"172.21.15.88:6805/1809528555","state":["exists","up"]},{"osd":2,"uuid":"3cd56229-2067-433a-b9f9-f47b1b878b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6801","nonce":3891687382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6803","nonce":3891687382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6807","nonce":3891687382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6805","nonce":3891687382}]},"public_addr":"172.21.15.192:6801/3891687382","cluster_addr":"172.21.15.192:6803/3891687382","heartbeat_back_addr":"172.21.15.192:6807/3891687382","heartbeat_front_addr":"172.21.15.192:6805/3891687382","state":["exists","up"]},{"osd":3,"uuid":"aae250ab-e242-4ef8-9c3a-a4fa36102c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6811","nonce":2920717314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6813","nonce":2920717314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6817","nonce":2920717314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6815","nonce":2920717314}]},"public_addr":"172.21.15.57:6811/2920717314","cluster_addr":"172.21.15.57:6813/2920717314","heartbeat_back_addr":"172.21.15.57:6817/2920717314","heartbeat_front_addr":"172.21.15.57:6815/2920717314","state":["exists","up"]},{"osd":4,"uuid":"8c8d1dfa-1cff-4f0a-ae72-22bd522ed696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6809","nonce":3911525610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6811","nonce":3911525610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6815","nonce":3911525610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6813","nonce":3911525610}]},"public_addr":"172.21.15.88:6809/3911525610","cluster_addr":"172.21.15.88:6811/3911525610","heartbeat_back_addr":"172.21.15.88:6815/3911525610","heartbeat_front_addr":"172.21.15.88:6813/3911525610","state":["exists","up"]},{"osd":5,"uuid":"5a2a56ab-2af9-4176-ba4d-b2138f0e7c75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6809","nonce":905436397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6811","nonce":905436397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6815","nonce":905436397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6813","nonce":905436397}]},"public_addr":"172.21.15.192:6809/905436397","cluster_addr":"172.21.15.192:6811/905436397","heartbeat_back_addr":"172.21.15.192:6815/905436397","heartbeat_front_addr":"172.21.15.192:6813/905436397","state":["exists","up"]},{"osd":6,"uuid":"03eafdda-278c-443c-b376-677eeb50262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6819","nonce":333894423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6821","nonce":333894423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6825","nonce":333894423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6823","nonce":333894423}]},"public_addr":"172.21.15.57:6819/333894423","cluster_addr":"172.21.15.57:6821/333894423","heartbeat_back_addr":"172.21.15.57:6825/333894423","heartbeat_front_addr":"172.21.15.57:6823/333894423","state":["exists","up"]},{"osd":7,"uuid":"3f01254e-be73-41c8-b979-3b6a7534e2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6817","nonce":3031485551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6819","nonce":3031485551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6823","nonce":3031485551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6821","nonce":3031485551}]},"public_addr":"172.21.15.88:6817/3031485551","cluster_addr":"172.21.15.88:6819/3031485551","heartbeat_back_addr":"172.21.15.88:6823/3031485551","heartbeat_front_addr":"172.21.15.88:6821/3031485551","state":["exists","up"]},{"osd":8,"uuid":"624a150e-f47b-4a76-8785-c818dd65342e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6817","nonce":918056390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6819","nonce":918056390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6823","nonce":918056390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6821","nonce":918056390}]},"public_addr":"172.21.15.192:6817/918056390","cluster_addr":"172.21.15.192:6819/918056390","heartbeat_back_addr":"172.21.15.192:6823/918056390","heartbeat_front_addr":"172.21.15.192:6821/918056390","state":["exists","up"]},{"osd":9,"uuid":"6d9cb69d-ed7b-4009-9ec5-027becc577d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6827","nonce":1773340763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6829","nonce":1773340763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6833","nonce":1773340763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6831","nonce":1773340763}]},"public_addr":"172.21.15.57:6827/1773340763","cluster_addr":"172.21.15.57:6829/1773340763","heartbeat_back_addr":"172.21.15.57:6833/1773340763","heartbeat_front_addr":"172.21.15.57:6831/1773340763","state":["exists","up"]},{"osd":10,"uuid":"d5799219-0979-40b1-a596-acd37f37a72d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6825","nonce":2781975579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6827","nonce":2781975579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6831","nonce":2781975579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6829","nonce":2781975579}]},"public_addr":"172.21.15.88:6825/2781975579","cluster_addr":"172.21.15.88:6827/2781975579","heartbeat_back_addr":"172.21.15.88:6831/2781975579","heartbeat_front_addr":"172.21.15.88:6829/2781975579","state":["exists","up"]},{"osd":11,"uuid":"e77cac6c-107e-41f9-8efd-e1140c4e3bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6825","nonce":2864190031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6827","nonce":2864190031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6831","nonce":2864190031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6829","nonce":2864190031}]},"public_addr":"172.21.15.192:6825/2864190031","cluster_addr":"172.21.15.192:6827/2864190031","heartbeat_back_addr":"172.21.15.192:6831/2864190031","heartbeat_front_addr":"172.21.15.192:6829/2864190031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:53:38.180971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:00.253039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:20.776122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:48.631094+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:09.288231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:29.595802+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:55.089126+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:15.637849+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:35.985101+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:01.544767+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:22.351743+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:43.255157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6801/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/3804015220":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6800/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1582727629":"2023-07-21T18:51:34.614904+0000","172.21.15.57:6800/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/2044626900":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/748511752":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1151100650":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6801/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/3446341201":"2023-07-21T18:52:02.610729+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T18:58:37.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 -- 172.21.15.57:0/2704814546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40540a11a0 msgr2=0x7f40540a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:37.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40540a11a0 0x7f40540a3660 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4058007c60 tx=0x7f40580073d0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 -- 172.21.15.57:0/2704814546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4068100780 msgr2=0x7f40681986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:37.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4068100780 0x7f40681986c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4064006010 tx=0x7f4064004730 comp rx=0 tx=0).stop 2023-07-20T18:58:37.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 -- 172.21.15.57:0/2704814546 shutdown_connections 2023-07-20T18:58:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40540a11a0 0x7f40540a3660 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4068102310 0x7f406819fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4068100780 0x7f40681986c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 --2- 172.21.15.57:0/2704814546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4068101970 0x7f4068198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 -- 172.21.15.57:0/2704814546 >> 172.21.15.57:0/2704814546 conn(0x7f40680fbcb0 msgr2=0x7f40681071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 -- 172.21.15.57:0/2704814546 shutdown_connections 2023-07-20T18:58:37.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.275+0000 7f40701ff700 1 -- 172.21.15.57:0/2704814546 wait complete. 2023-07-20T18:58:37.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:36 smithi192 ceph-mon[129200]: osdmap e86: 12 total, 12 up, 12 in 2023-07-20T18:58:37.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:36 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3340791297' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-20T18:58:37.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:36 smithi192 ceph-mon[129200]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:58:37.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:36 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2886114833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:37.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:36 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/832109962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:37.362 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T18:58:37.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:36 smithi088 ceph-mon[128823]: osdmap e86: 12 total, 12 up, 12 in 2023-07-20T18:58:37.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:36 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3340791297' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-20T18:58:37.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:36 smithi088 ceph-mon[128823]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:58:37.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:36 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2886114833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:37.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:36 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/832109962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:37.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.515+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/2725366399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e64100770 msgr2=0x7f6e64100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:37.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.515+0000 7f6e6b9ff700 1 --2- 172.21.15.57:0/2725366399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e64100770 0x7f6e64100b90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6e600099c0 tx=0x7f6e60009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.515+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/2725366399 shutdown_connections 2023-07-20T18:58:37.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.515+0000 7f6e6b9ff700 1 --2- 172.21.15.57:0/2725366399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e64102300 0x7f6e64108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.515+0000 7f6e6b9ff700 1 --2- 172.21.15.57:0/2725366399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e64100770 0x7f6e64100b90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.515+0000 7f6e6b9ff700 1 --2- 172.21.15.57:0/2725366399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e64101960 0x7f6e64101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.515+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/2725366399 >> 172.21.15.57:0/2725366399 conn(0x7f6e640fbca0 msgr2=0x7f6e640fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:37.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.515+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/2725366399 shutdown_connections 2023-07-20T18:58:37.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.515+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/2725366399 wait complete. 2023-07-20T18:58:37.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.516+0000 7f6e6b9ff700 1 Processor -- start 2023-07-20T18:58:37.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.516+0000 7f6e6b9ff700 1 -- start start 2023-07-20T18:58:37.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.516+0000 7f6e6b9ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e64100770 0x7f6e64074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:37.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.516+0000 7f6e6b9ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e64101960 0x7f6e64072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:37.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e6b9ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e64102300 0x7f6e640730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:37.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e6b9ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e64073840 con 0x7f6e64102300 2023-07-20T18:58:37.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e6b9ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640739b0 con 0x7f6e64101960 2023-07-20T18:58:37.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e6b9ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e641a0560 con 0x7f6e64100770 2023-07-20T18:58:37.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e69f9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e64102300 0x7f6e640730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:37.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e6979b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e64100770 0x7f6e64074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:37.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e68f9a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e64101960 0x7f6e64072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:37.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e69f9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e64102300 0x7f6e640730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44750/0 (socket says 172.21.15.57:44750) 2023-07-20T18:58:37.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e69f9c700 1 -- 172.21.15.57:0/1035689601 learned_addr learned my addr 172.21.15.57:0/1035689601 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:37.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e69f9c700 1 -- 172.21.15.57:0/1035689601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e64100770 msgr2=0x7f6e64074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:37.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e69f9c700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e64100770 0x7f6e64074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e69f9c700 1 -- 172.21.15.57:0/1035689601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e64101960 msgr2=0x7f6e64072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:37.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e69f9c700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e64101960 0x7f6e64072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e69f9c700 1 -- 172.21.15.57:0/1035689601 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e600095f0 con 0x7f6e64102300 2023-07-20T18:58:37.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.517+0000 7f6e69f9c700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e64102300 0x7f6e640730a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00ef30 tx=0x7f6e5c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:37.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.518+0000 7f6e5a7fc700 1 -- 172.21.15.57:0/1035689601 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c018070 con 0x7f6e64102300 2023-07-20T18:58:37.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.518+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/1035689601 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e641a07e0 con 0x7f6e64102300 2023-07-20T18:58:37.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.518+0000 7f6e5a7fc700 1 -- 172.21.15.57:0/1035689601 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f6e5c00f040 con 0x7f6e64102300 2023-07-20T18:58:37.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.518+0000 7f6e5a7fc700 1 -- 172.21.15.57:0/1035689601 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c00cd50 con 0x7f6e64102300 2023-07-20T18:58:37.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.518+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/1035689601 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e641a0cf0 con 0x7f6e64102300 2023-07-20T18:58:37.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.519+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/1035689601 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e64107600 con 0x7f6e64102300 2023-07-20T18:58:37.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.521+0000 7f6e5a7fc700 1 -- 172.21.15.57:0/1035689601 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6e5c003a00 con 0x7f6e64102300 2023-07-20T18:58:37.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.522+0000 7f6e5a7fc700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6e500a11a0 0x7f6e500a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:37.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.522+0000 7f6e5a7fc700 1 -- 172.21.15.57:0/1035689601 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6e5c0cd060 con 0x7f6e64102300 2023-07-20T18:58:37.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.522+0000 7f6e6979b700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6e500a11a0 0x7f6e500a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:37.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.523+0000 7f6e6979b700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6e500a11a0 0x7f6e500a3660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000b580 tx=0x7f6e60005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:37.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.524+0000 7f6e5a7fc700 1 -- 172.21.15.57:0/1035689601 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6e5c0965f0 con 0x7f6e64102300 2023-07-20T18:58:37.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.779+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/1035689601 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e6402c970 con 0x7f6e64102300 2023-07-20T18:58:37.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.780+0000 7f6e5a7fc700 1 -- 172.21.15.57:0/1035689601 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10763 (secure 0 0 0) 0x7f6e5c099c10 con 0x7f6e64102300 2023-07-20T18:58:37.830 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:37.831 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14445,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4001514318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4001514318},{"type":"v1","addr":"172.21.15.57:6835","nonce":4001514318}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/3251478878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":3251478878},{"type":"v1","addr":"172.21.15.57:6837","nonce":3251478878}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14502,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1597260444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1597260444},{"type":"v1","addr":"172.21.15.57:6839","nonce":1597260444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/1188419010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":1188419010},{"type":"v1","addr":"172.21.15.57:6841","nonce":1188419010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/433531829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":433531829},{"type":"v1","addr":"172.21.15.192:6833","nonce":433531829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/749062861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":749062861},{"type":"v1","addr":"172.21.15.88:6833","nonce":749062861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2219652585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2219652585},{"type":"v1","addr":"172.21.15.192:6835","nonce":2219652585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/1373743783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1373743783},{"type":"v1","addr":"172.21.15.88:6835","nonce":1373743783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2391717255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2391717255},{"type":"v1","addr":"172.21.15.192:6837","nonce":2391717255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/678288704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":678288704},{"type":"v1","addr":"172.21.15.88:6837","nonce":678288704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/1456232015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1456232015},{"type":"v1","addr":"172.21.15.192:6839","nonce":1456232015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-20T18:58:30.332763+0000","modified":"2023-07-20T18:58:35.858060+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24566},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/2098157844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2098157844},{"type":"v1","addr":"172.21.15.88:6839","nonce":2098157844}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-20T18:58:37.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/1035689601 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6e500a11a0 msgr2=0x7f6e500a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:37.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6e500a11a0 0x7f6e500a3660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000b580 tx=0x7f6e60005fb0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/1035689601 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e64102300 msgr2=0x7f6e640730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:37.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e64102300 0x7f6e640730a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00ef30 tx=0x7f6e5c00c3a0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/1035689601 shutdown_connections 2023-07-20T18:58:37.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f6e500a11a0 0x7f6e500a3660 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e64100770 0x7f6e64074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e64102300 0x7f6e640730a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 --2- 172.21.15.57:0/1035689601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e64101960 0x7f6e64072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:37.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/1035689601 >> 172.21.15.57:0/1035689601 conn(0x7f6e640fbca0 msgr2=0x7f6e640fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:37.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/1035689601 shutdown_connections 2023-07-20T18:58:37.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:37.783+0000 7f6e6b9ff700 1 -- 172.21.15.57:0/1035689601 wait complete. 2023-07-20T18:58:37.836 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 17 2023-07-20T18:58:37.851 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T18:58:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.008+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1884113548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ea4101960 msgr2=0x7f3ea4101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.008+0000 7f3eab7ff700 1 --2- 172.21.15.57:0/1884113548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ea4101960 0x7f3ea4101dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3e940099c0 tx=0x7f3e94009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.008+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1884113548 shutdown_connections 2023-07-20T18:58:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.008+0000 7f3eab7ff700 1 --2- 172.21.15.57:0/1884113548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea4102300 0x7f3ea4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.008+0000 7f3eab7ff700 1 --2- 172.21.15.57:0/1884113548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ea4100770 0x7f3ea4100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.008+0000 7f3eab7ff700 1 --2- 172.21.15.57:0/1884113548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ea4101960 0x7f3ea4101dc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.008+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1884113548 >> 172.21.15.57:0/1884113548 conn(0x7f3ea40fbca0 msgr2=0x7f3ea40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.009+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1884113548 shutdown_connections 2023-07-20T18:58:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.009+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1884113548 wait complete. 2023-07-20T18:58:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.009+0000 7f3eab7ff700 1 Processor -- start 2023-07-20T18:58:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3eab7ff700 1 -- start start 2023-07-20T18:58:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3eab7ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea4100770 0x7f3ea4196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3eab7ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ea4101960 0x7f3ea4196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3eab7ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ea4102300 0x7f3ea419dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3eab7ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea419e080 con 0x7f3ea4102300 2023-07-20T18:58:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3eab7ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea419e1c0 con 0x7f3ea4101960 2023-07-20T18:58:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3eab7ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea419e350 con 0x7f3ea4100770 2023-07-20T18:58:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3ea9d9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ea4102300 0x7f3ea419dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3ea9d9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ea4102300 0x7f3ea419dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44776/0 (socket says 172.21.15.57:44776) 2023-07-20T18:58:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3ea9d9c700 1 -- 172.21.15.57:0/1477724963 learned_addr learned my addr 172.21.15.57:0/1477724963 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3ea959b700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea4100770 0x7f3ea4196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.010+0000 7f3ea8d9a700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ea4101960 0x7f3ea4196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3ea9d9c700 1 -- 172.21.15.57:0/1477724963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea4100770 msgr2=0x7f3ea4196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3ea9d9c700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea4100770 0x7f3ea4196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3ea9d9c700 1 -- 172.21.15.57:0/1477724963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ea4101960 msgr2=0x7f3ea4196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3ea9d9c700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ea4101960 0x7f3ea4196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3ea9d9c700 1 -- 172.21.15.57:0/1477724963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea0009520 con 0x7f3ea4102300 2023-07-20T18:58:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3ea9d9c700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ea4102300 0x7f3ea419dab0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c007ee0 tx=0x7f3e9c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3e9a7fc700 1 -- 172.21.15.57:0/1477724963 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c017070 con 0x7f3ea4102300 2023-07-20T18:58:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3e9a7fc700 1 -- 172.21.15.57:0/1477724963 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f3e9c00f040 con 0x7f3ea4102300 2023-07-20T18:58:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1477724963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e940095f0 con 0x7f3ea4102300 2023-07-20T18:58:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3e9a7fc700 1 -- 172.21.15.57:0/1477724963 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c005610 con 0x7f3ea4102300 2023-07-20T18:58:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.011+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1477724963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea419e9b0 con 0x7f3ea4102300 2023-07-20T18:58:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.012+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1477724963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea4107c90 con 0x7f3ea4102300 2023-07-20T18:58:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.014+0000 7f3e9a7fc700 1 -- 172.21.15.57:0/1477724963 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3e9c005770 con 0x7f3ea4102300 2023-07-20T18:58:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.016+0000 7f3e9a7fc700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e900a1220 0x7f3e900a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.016+0000 7f3e9a7fc700 1 -- 172.21.15.57:0/1477724963 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3e9c0cc8a0 con 0x7f3ea4102300 2023-07-20T18:58:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.016+0000 7f3ea959b700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e900a1220 0x7f3e900a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.016+0000 7f3ea959b700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e900a1220 0x7f3e900a36e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ea41017c0 tx=0x7f3ea0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:38.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.018+0000 7f3e9a7fc700 1 -- 172.21.15.57:0/1477724963 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3e9c095da0 con 0x7f3ea4102300 2023-07-20T18:58:38.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:37 smithi057 ceph-mon[126340]: pgmap v273: 129 pgs: 4 creating+peering, 115 active+clean, 10 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2023-07-20T18:58:38.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:37 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2704814546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:38.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:37 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1035689601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:38.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.228+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1477724963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ea402cbc0 con 0x7f3ea4102300 2023-07-20T18:58:38.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.230+0000 7f3e9a7fc700 1 -- 172.21.15.57:0/1477724963 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23911 (secure 0 0 0) 0x7f3e9c009390 con 0x7f3ea4102300 2023-07-20T18:58:38.279 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:38.280 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":86,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","created":"2023-07-20T18:51:01.245641+0000","modified":"2023-07-20T18:58:35.383850+0000","last_up_change":"2023-07-20T18:57:44.823544+0000","last_in_change":"2023-07-20T18:57:27.428782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T18:54:23.215777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T18:58:28.447992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T18:58:29.772740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T18:58:33.395093+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"627a67ea-1366-4b3f-babd-067f75bad1e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6803","nonce":3023441607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6805","nonce":3023441607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6809","nonce":3023441607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6807","nonce":3023441607}]},"public_addr":"172.21.15.57:6803/3023441607","cluster_addr":"172.21.15.57:6805/3023441607","heartbeat_back_addr":"172.21.15.57:6809/3023441607","heartbeat_front_addr":"172.21.15.57:6807/3023441607","state":["exists","up"]},{"osd":1,"uuid":"adcdb236-334e-4478-98a8-7d0628492082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6801","nonce":1809528555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6803","nonce":1809528555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6807","nonce":1809528555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6805","nonce":1809528555}]},"public_addr":"172.21.15.88:6801/1809528555","cluster_addr":"172.21.15.88:6803/1809528555","heartbeat_back_addr":"172.21.15.88:6807/1809528555","heartbeat_front_addr":"172.21.15.88:6805/1809528555","state":["exists","up"]},{"osd":2,"uuid":"3cd56229-2067-433a-b9f9-f47b1b878b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6801","nonce":3891687382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6803","nonce":3891687382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6807","nonce":3891687382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6805","nonce":3891687382}]},"public_addr":"172.21.15.192:6801/3891687382","cluster_addr":"172.21.15.192:6803/3891687382","heartbeat_back_addr":"172.21.15.192:6807/3891687382","heartbeat_front_addr":"172.21.15.192:6805/3891687382","state":["exists","up"]},{"osd":3,"uuid":"aae250ab-e242-4ef8-9c3a-a4fa36102c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6811","nonce":2920717314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6813","nonce":2920717314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6817","nonce":2920717314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6815","nonce":2920717314}]},"public_addr":"172.21.15.57:6811/2920717314","cluster_addr":"172.21.15.57:6813/2920717314","heartbeat_back_addr":"172.21.15.57:6817/2920717314","heartbeat_front_addr":"172.21.15.57:6815/2920717314","state":["exists","up"]},{"osd":4,"uuid":"8c8d1dfa-1cff-4f0a-ae72-22bd522ed696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6809","nonce":3911525610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6811","nonce":3911525610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6815","nonce":3911525610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6813","nonce":3911525610}]},"public_addr":"172.21.15.88:6809/3911525610","cluster_addr":"172.21.15.88:6811/3911525610","heartbeat_back_addr":"172.21.15.88:6815/3911525610","heartbeat_front_addr":"172.21.15.88:6813/3911525610","state":["exists","up"]},{"osd":5,"uuid":"5a2a56ab-2af9-4176-ba4d-b2138f0e7c75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6809","nonce":905436397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6811","nonce":905436397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6815","nonce":905436397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6813","nonce":905436397}]},"public_addr":"172.21.15.192:6809/905436397","cluster_addr":"172.21.15.192:6811/905436397","heartbeat_back_addr":"172.21.15.192:6815/905436397","heartbeat_front_addr":"172.21.15.192:6813/905436397","state":["exists","up"]},{"osd":6,"uuid":"03eafdda-278c-443c-b376-677eeb50262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6819","nonce":333894423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6821","nonce":333894423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6825","nonce":333894423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6823","nonce":333894423}]},"public_addr":"172.21.15.57:6819/333894423","cluster_addr":"172.21.15.57:6821/333894423","heartbeat_back_addr":"172.21.15.57:6825/333894423","heartbeat_front_addr":"172.21.15.57:6823/333894423","state":["exists","up"]},{"osd":7,"uuid":"3f01254e-be73-41c8-b979-3b6a7534e2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6817","nonce":3031485551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6819","nonce":3031485551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6823","nonce":3031485551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6821","nonce":3031485551}]},"public_addr":"172.21.15.88:6817/3031485551","cluster_addr":"172.21.15.88:6819/3031485551","heartbeat_back_addr":"172.21.15.88:6823/3031485551","heartbeat_front_addr":"172.21.15.88:6821/3031485551","state":["exists","up"]},{"osd":8,"uuid":"624a150e-f47b-4a76-8785-c818dd65342e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6817","nonce":918056390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6819","nonce":918056390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6823","nonce":918056390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6821","nonce":918056390}]},"public_addr":"172.21.15.192:6817/918056390","cluster_addr":"172.21.15.192:6819/918056390","heartbeat_back_addr":"172.21.15.192:6823/918056390","heartbeat_front_addr":"172.21.15.192:6821/918056390","state":["exists","up"]},{"osd":9,"uuid":"6d9cb69d-ed7b-4009-9ec5-027becc577d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6827","nonce":1773340763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6829","nonce":1773340763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6833","nonce":1773340763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6831","nonce":1773340763}]},"public_addr":"172.21.15.57:6827/1773340763","cluster_addr":"172.21.15.57:6829/1773340763","heartbeat_back_addr":"172.21.15.57:6833/1773340763","heartbeat_front_addr":"172.21.15.57:6831/1773340763","state":["exists","up"]},{"osd":10,"uuid":"d5799219-0979-40b1-a596-acd37f37a72d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6825","nonce":2781975579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6827","nonce":2781975579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6831","nonce":2781975579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6829","nonce":2781975579}]},"public_addr":"172.21.15.88:6825/2781975579","cluster_addr":"172.21.15.88:6827/2781975579","heartbeat_back_addr":"172.21.15.88:6831/2781975579","heartbeat_front_addr":"172.21.15.88:6829/2781975579","state":["exists","up"]},{"osd":11,"uuid":"e77cac6c-107e-41f9-8efd-e1140c4e3bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6825","nonce":2864190031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6827","nonce":2864190031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6831","nonce":2864190031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6829","nonce":2864190031}]},"public_addr":"172.21.15.192:6825/2864190031","cluster_addr":"172.21.15.192:6827/2864190031","heartbeat_back_addr":"172.21.15.192:6831/2864190031","heartbeat_front_addr":"172.21.15.192:6829/2864190031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:53:38.180971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:00.253039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:20.776122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:48.631094+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:09.288231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:29.595802+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:55.089126+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:15.637849+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:35.985101+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:01.544767+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:22.351743+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:43.255157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6801/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/3804015220":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6800/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1582727629":"2023-07-21T18:51:34.614904+0000","172.21.15.57:6800/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/2044626900":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/748511752":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1151100650":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6801/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/3446341201":"2023-07-21T18:52:02.610729+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T18:58:38.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1477724963 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e900a1220 msgr2=0x7f3e900a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e900a1220 0x7f3e900a36e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ea41017c0 tx=0x7f3ea0009260 comp rx=0 tx=0).stop 2023-07-20T18:58:38.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1477724963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ea4102300 msgr2=0x7f3ea419dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ea4102300 0x7f3ea419dab0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c007ee0 tx=0x7f3e9c00d3b0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1477724963 shutdown_connections 2023-07-20T18:58:38.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f3e900a1220 0x7f3e900a36e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea4100770 0x7f3ea4196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ea4102300 0x7f3ea419dab0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 --2- 172.21.15.57:0/1477724963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ea4101960 0x7f3ea4196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1477724963 >> 172.21.15.57:0/1477724963 conn(0x7f3ea40fbca0 msgr2=0x7f3ea4104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:38.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1477724963 shutdown_connections 2023-07-20T18:58:38.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.232+0000 7f3eab7ff700 1 -- 172.21.15.57:0/1477724963 wait complete. 2023-07-20T18:58:38.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-07-20T18:58:30.332763+0000', 'modified': '2023-07-20T18:58:35.858060+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24566}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24566': {'gid': 24566, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.88:6839/2098157844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6838', 'nonce': 2098157844}, {'type': 'v1', 'addr': '172.21.15.88:6839', 'nonce': 2098157844}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-07-20T18:58:38.301 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-07-20T18:58:38.301 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-20T18:58:38.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:37 smithi192 ceph-mon[129200]: pgmap v273: 129 pgs: 4 creating+peering, 115 active+clean, 10 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2023-07-20T18:58:38.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:37 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2704814546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:38.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:37 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1035689601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:38.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:37 smithi088 ceph-mon[128823]: pgmap v273: 129 pgs: 4 creating+peering, 115 active+clean, 10 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2023-07-20T18:58:38.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:37 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2704814546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:38.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:37 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1035689601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T18:58:38.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.457+0000 7f861cfff700 1 -- 172.21.15.57:0/3168523509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8618074750 msgr2=0x7f8618072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.457+0000 7f861cfff700 1 --2- 172.21.15.57:0/3168523509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8618074750 0x7f8618072ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8608009990 tx=0x7f8608009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.457+0000 7f861cfff700 1 -- 172.21.15.57:0/3168523509 shutdown_connections 2023-07-20T18:58:38.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.457+0000 7f861cfff700 1 --2- 172.21.15.57:0/3168523509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8618108630 0x7f861810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.457+0000 7f861cfff700 1 --2- 172.21.15.57:0/3168523509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8618074750 0x7f8618072ba0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.457+0000 7f861cfff700 1 --2- 172.21.15.57:0/3168523509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8618073170 0x7f86180735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.457+0000 7f861cfff700 1 -- 172.21.15.57:0/3168523509 >> 172.21.15.57:0/3168523509 conn(0x7f86180fbcb0 msgr2=0x7f86180fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:38.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.458+0000 7f861cfff700 1 -- 172.21.15.57:0/3168523509 shutdown_connections 2023-07-20T18:58:38.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.458+0000 7f861cfff700 1 -- 172.21.15.57:0/3168523509 wait complete. 2023-07-20T18:58:38.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.458+0000 7f861cfff700 1 Processor -- start 2023-07-20T18:58:38.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f861cfff700 1 -- start start 2023-07-20T18:58:38.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f861cfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8618073170 0x7f8618198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f861cfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8618074750 0x7f8618198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f861cfff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8618108630 0x7f861819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f861cfff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86181a02a0 con 0x7f8618074750 2023-07-20T18:58:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f861cfff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86181a03e0 con 0x7f8618073170 2023-07-20T18:58:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f861cfff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86181a0570 con 0x7f8618108630 2023-07-20T18:58:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f8615d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8618074750 0x7f8618198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f8615d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8618074750 0x7f8618198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44794/0 (socket says 172.21.15.57:44794) 2023-07-20T18:58:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f8615d9b700 1 -- 172.21.15.57:0/3444065558 learned_addr learned my addr 172.21.15.57:0/3444065558 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f861659c700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8618073170 0x7f8618198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.459+0000 7f8616d9d700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8618108630 0x7f861819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f8615d9b700 1 -- 172.21.15.57:0/3444065558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8618108630 msgr2=0x7f861819fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f8615d9b700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8618108630 0x7f861819fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f8615d9b700 1 -- 172.21.15.57:0/3444065558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8618073170 msgr2=0x7f8618198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f8615d9b700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8618073170 0x7f8618198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f8615d9b700 1 -- 172.21.15.57:0/3444065558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c009520 con 0x7f8618074750 2023-07-20T18:58:38.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f8615d9b700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8618074750 0x7f8618198c60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f860c00eb30 tx=0x7f860c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f86077fe700 1 -- 172.21.15.57:0/3444065558 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c009880 con 0x7f8618074750 2023-07-20T18:58:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f86077fe700 1 -- 172.21.15.57:0/3444065558 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f860c004500 con 0x7f8618074750 2023-07-20T18:58:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f861cfff700 1 -- 172.21.15.57:0/3444065558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86080095f0 con 0x7f8618074750 2023-07-20T18:58:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f86077fe700 1 -- 172.21.15.57:0/3444065558 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c003e70 con 0x7f8618074750 2023-07-20T18:58:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.460+0000 7f861cfff700 1 -- 172.21.15.57:0/3444065558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86181a0bd0 con 0x7f8618074750 2023-07-20T18:58:38.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.463+0000 7f86077fe700 1 -- 172.21.15.57:0/3444065558 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f860c00cbd0 con 0x7f8618074750 2023-07-20T18:58:38.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.465+0000 7f86308eab80 1 Processor -- start 2023-07-20T18:58:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.465+0000 7f86308eab80 1 -- start start 2023-07-20T18:58:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.465+0000 7f86077fe700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f86000a1150 0x7f86000a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.465+0000 7f86077fe700 1 -- 172.21.15.57:0/3444065558 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f860c014070 con 0x7f8618074750 2023-07-20T18:58:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.465+0000 7f8616d9d700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f86000a1150 0x7f86000a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.465+0000 7f86308eab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1ada540 0x55b6a1ada960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f86308eab80 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1adb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f86308eab80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1adb900 0x55b6a1adbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f86308eab80 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a1a55d70 con 0x55b6a1adaea0 2023-07-20T18:58:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f86308eab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a1a32540 con 0x55b6a1ada540 2023-07-20T18:58:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f86308eab80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a1a2fe60 con 0x55b6a1adb900 2023-07-20T18:58:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8615d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1adb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8615d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1adb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44806/0 (socket says 172.21.15.57:44806) 2023-07-20T18:58:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8615d9b700 1 -- 172.21.15.57:0/2199318531 learned_addr learned my addr 172.21.15.57:0/2199318531 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f861659c700 1 --2- 172.21.15.57:0/2199318531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1ada540 0x55b6a1ada960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8616d9d700 1 --2- 172.21.15.57:0/2199318531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1adb900 0x55b6a1adbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8615d9b700 1 -- 172.21.15.57:0/2199318531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1adb900 msgr2=0x55b6a1adbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T18:58:38.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8615d9b700 1 --2- 172.21.15.57:0/2199318531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1adb900 0x55b6a1adbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8615d9b700 1 -- 172.21.15.57:0/2199318531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1ada540 msgr2=0x55b6a1ada960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T18:58:38.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8615d9b700 1 --2- 172.21.15.57:0/2199318531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1ada540 0x55b6a1ada960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8616d9d700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f86000a1150 0x7f86000a3610 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f861000f590 tx=0x7f861000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:38.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8615d9b700 1 -- 172.21.15.57:0/2199318531 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6a18ac5e0 con 0x55b6a1adaea0 2023-07-20T18:58:38.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.466+0000 7f8615d9b700 1 --2- 172.21.15.57:0/2199318531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1adb320 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f860c0a2070 tx=0x7f860c099fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad69dfd34e3d7f7 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:38.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.467+0000 7f8604ff9700 1 -- 172.21.15.57:0/2199318531 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c099ab0 con 0x55b6a1adaea0 2023-07-20T18:58:38.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.467+0000 7f8604ff9700 1 -- 172.21.15.57:0/2199318531 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f860c01e2e0 con 0x55b6a1adaea0 2023-07-20T18:58:38.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.467+0000 7f8604ff9700 1 -- 172.21.15.57:0/2199318531 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c0a2110 con 0x55b6a1adaea0 2023-07-20T18:58:38.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.467+0000 7f86308eab80 1 -- 172.21.15.57:0/2199318531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 msgr2=0x55b6a1adb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.467+0000 7f86308eab80 1 --2- 172.21.15.57:0/2199318531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1adb320 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f860c0a2070 tx=0x7f860c099fa0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.467+0000 7f86308eab80 1 -- 172.21.15.57:0/2199318531 shutdown_connections 2023-07-20T18:58:38.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.467+0000 7f86308eab80 1 --2- 172.21.15.57:0/2199318531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1adb900 0x55b6a1adbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.467+0000 7f86308eab80 1 --2- 172.21.15.57:0/2199318531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1ada540 0x55b6a1ada960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.467+0000 7f86308eab80 1 --2- 172.21.15.57:0/2199318531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1adb320 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.467+0000 7f86308eab80 1 -- 172.21.15.57:0/2199318531 >> 172.21.15.57:0/2199318531 conn(0x55b6a1a37c80 msgr2=0x55b6a1ad5230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:38.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.468+0000 7f86308eab80 1 -- 172.21.15.57:0/2199318531 shutdown_connections 2023-07-20T18:58:38.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.468+0000 7f86308eab80 1 -- 172.21.15.57:0/2199318531 wait complete. 2023-07-20T18:58:38.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.468+0000 7f86308eab80 1 -- start start 2023-07-20T18:58:38.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.468+0000 7f86308eab80 1 Processor -- start 2023-07-20T18:58:38.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f86308eab80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1ada540 0x55b6a1ae3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f86308eab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adaea0 0x55b6a1ae4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f86308eab80 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adb900 0x55b6a1aeb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f86308eab80 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a1a2b750 con 0x55b6a1adb900 2023-07-20T18:58:38.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f86308eab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a19caa70 con 0x55b6a1adaea0 2023-07-20T18:58:38.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f86308eab80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a19ca770 con 0x55b6a1ada540 2023-07-20T18:58:38.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f86308eab80 10 client.0 _ll_register_callbacks cb 0x55b6a19c8630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-20T18:58:38.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f8616d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adb900 0x55b6a1aeb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f8615d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adaea0 0x55b6a1ae4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f8616d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adb900 0x55b6a1aeb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44822/0 (socket says 172.21.15.57:44822) 2023-07-20T18:58:38.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f8616d9d700 1 -- 172.21.15.57:0/4158655600 learned_addr learned my addr 172.21.15.57:0/4158655600 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:38.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f861659c700 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1ada540 0x55b6a1ae3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f8616d9d700 1 -- 172.21.15.57:0/4158655600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1ada540 msgr2=0x55b6a1ae3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T18:58:38.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f8616d9d700 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1ada540 0x55b6a1ae3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.469+0000 7f8616d9d700 1 -- 172.21.15.57:0/4158655600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adaea0 msgr2=0x55b6a1ae4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T18:58:38.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f8616d9d700 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adaea0 0x55b6a1ae4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f8616d9d700 1 -- 172.21.15.57:0/4158655600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86181a0bd0 con 0x55b6a1adb900 2023-07-20T18:58:38.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f8615d9b700 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adaea0 0x55b6a1ae4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:38.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f8616d9d700 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adb900 0x55b6a1aeb4c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8610017e50 tx=0x7f8610004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b666670524210d server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:38.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f85fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-07-20T18:58:38.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f85fdffb700 1 -- 172.21.15.57:0/4158655600 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861000e040 con 0x55b6a1adb900 2023-07-20T18:58:38.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f86308eab80 1 -- 172.21.15.57:0/4158655600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b6a1acd420 con 0x55b6a1adb900 2023-07-20T18:58:38.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f85fdffb700 1 -- 172.21.15.57:0/4158655600 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f861001c040 con 0x55b6a1adb900 2023-07-20T18:58:38.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f85fdffb700 1 -- 172.21.15.57:0/4158655600 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8610015d90 con 0x55b6a1adb900 2023-07-20T18:58:38.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f86308eab80 10 client.14628 fetch_fsmap learned FSMap version 17 2023-07-20T18:58:38.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.470+0000 7f86308eab80 1 -- 172.21.15.57:0/4158655600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55b6a1858d90 con 0x55b6a1adb900 2023-07-20T18:58:38.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.471+0000 7f85fdffb700 1 -- 172.21.15.57:0/4158655600 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x55b6a1acd420 con 0x55b6a1adb900 2023-07-20T18:58:38.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.471+0000 7f85fdffb700 1 -- 172.21.15.57:0/4158655600 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f8610004620 con 0x55b6a1adb900 2023-07-20T18:58:38.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.471+0000 7f86308eab80 10 client.14628 fetch_fsmap finished waiting for FSMap version 17 2023-07-20T18:58:38.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.471+0000 7f86308eab80 10 client.14628 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-20T18:58:38.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.471+0000 7f86308eab80 20 client.14628 populate_metadata read hostname 'smithi057' 2023-07-20T18:58:38.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.471+0000 7f86308eab80 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] conn(0x55b6a1aef1f0 0x55b6a1aee620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.471+0000 7f86308eab80 4 client.14628 mds_command: new command op to 24566 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-20T18:58:38.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.471+0000 7f86308eab80 1 -- 172.21.15.57:0/4158655600 --> [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b6a1aa9980 con 0x55b6a1aef1f0 2023-07-20T18:58:38.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.471+0000 7f861659c700 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] conn(0x55b6a1aef1f0 0x55b6a1aee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.472+0000 7f861659c700 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] conn(0x55b6a1aef1f0 0x55b6a1aee620 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ca26bc6841b728e server_cookie=b0cc793a3d26bcf4 in_seq=0 out_seq=0 2023-07-20T18:58:38.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.472+0000 7f85fdffb700 10 client.14628 ms_handle_connect on v2:172.21.15.88:6838/2098157844 2023-07-20T18:58:38.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f85fdffb700 1 -- 172.21.15.57:0/4158655600 <== mds.0 v2:172.21.15.88:6838/2098157844 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55b6a1aa9980 con 0x55b6a1aef1f0 2023-07-20T18:58:38.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f85fdffb700 10 client.14628 handle_command_reply: tid=0 2023-07-20T18:58:38.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 client.14628 shutdown 2023-07-20T18:58:38.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 -- 172.21.15.57:0/4158655600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adb900 msgr2=0x55b6a1aeb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adb900 0x55b6a1aeb4c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8610017e50 tx=0x7f8610004ad0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 -- 172.21.15.57:0/4158655600 shutdown_connections 2023-07-20T18:58:38.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] conn(0x55b6a1aef1f0 0x55b6a1aee620 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1ada540 0x55b6a1ae3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adb900 0x55b6a1aeb4c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f85fdffb700 0 client.14628 ms_handle_reset on v2:172.21.15.88:6838/2098157844 2023-07-20T18:58:38.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 --2- 172.21.15.57:0/4158655600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adaea0 0x55b6a1ae4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 -- 172.21.15.57:0/4158655600 >> 172.21.15.57:0/4158655600 conn(0x55b6a1a37c80 msgr2=0x55b6a1ad4930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:38.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 -- 172.21.15.57:0/4158655600 shutdown_connections 2023-07-20T18:58:38.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 1 -- 172.21.15.57:0/4158655600 wait complete. 2023-07-20T18:58:38.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.473+0000 7f86308eab80 20 client.14628 trim_cache size 0 max 16384 2023-07-20T18:58:38.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.491+0000 7f86308eab80 1 Processor -- start 2023-07-20T18:58:38.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.491+0000 7f86308eab80 1 -- start start 2023-07-20T18:58:38.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f86308eab80 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1ada540 0x55b6a1aee620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f86308eab80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1adaea0 0x55b6a1af7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f86308eab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adb900 0x55b6a1ade780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f86308eab80 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a1ae3b10 con 0x55b6a1ada540 2023-07-20T18:58:38.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f86308eab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a18ae8d0 con 0x55b6a1adb900 2023-07-20T18:58:38.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f86308eab80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a1ad1f30 con 0x55b6a1adaea0 2023-07-20T18:58:38.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f861659c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1ada540 0x55b6a1aee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f861659c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1ada540 0x55b6a1aee620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44836/0 (socket says 172.21.15.57:44836) 2023-07-20T18:58:38.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f861659c700 1 -- 172.21.15.57:0/3202621065 learned_addr learned my addr 172.21.15.57:0/3202621065 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:38.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f8616d9d700 1 --2- 172.21.15.57:0/3202621065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adb900 0x55b6a1ade780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f8615d9b700 1 --2- 172.21.15.57:0/3202621065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1adaea0 0x55b6a1af7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f861659c700 1 -- 172.21.15.57:0/3202621065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1adaea0 msgr2=0x55b6a1af7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T18:58:38.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f861659c700 1 --2- 172.21.15.57:0/3202621065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1adaea0 0x55b6a1af7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f861659c700 1 -- 172.21.15.57:0/3202621065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adb900 msgr2=0x55b6a1ade780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T18:58:38.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f861659c700 1 --2- 172.21.15.57:0/3202621065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adb900 0x55b6a1ade780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.492+0000 7f861659c700 1 -- 172.21.15.57:0/3202621065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6a1841c10 con 0x55b6a1ada540 2023-07-20T18:58:38.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f861659c700 1 --2- 172.21.15.57:0/3202621065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1ada540 0x55b6a1aee620 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8608009990 tx=0x7f8608005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92cb7e70908a8d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:38.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f86057fa700 1 -- 172.21.15.57:0/3202621065 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860801d040 con 0x55b6a1ada540 2023-07-20T18:58:38.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f86057fa700 1 -- 172.21.15.57:0/3202621065 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f860801e040 con 0x55b6a1ada540 2023-07-20T18:58:38.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f86057fa700 1 -- 172.21.15.57:0/3202621065 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86080054d0 con 0x55b6a1ada540 2023-07-20T18:58:38.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f86308eab80 1 -- 172.21.15.57:0/3202621065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1ada540 msgr2=0x55b6a1aee620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f86308eab80 1 --2- 172.21.15.57:0/3202621065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1ada540 0x55b6a1aee620 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8608009990 tx=0x7f8608005fb0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f86308eab80 1 -- 172.21.15.57:0/3202621065 shutdown_connections 2023-07-20T18:58:38.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f86308eab80 1 --2- 172.21.15.57:0/3202621065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adb900 0x55b6a1ade780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f86308eab80 1 --2- 172.21.15.57:0/3202621065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1ada540 0x55b6a1aee620 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f86308eab80 1 --2- 172.21.15.57:0/3202621065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1adaea0 0x55b6a1af7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.493+0000 7f86308eab80 1 -- 172.21.15.57:0/3202621065 >> 172.21.15.57:0/3202621065 conn(0x55b6a1a37c80 msgr2=0x55b6a1ae2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:38.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.494+0000 7f86308eab80 1 -- 172.21.15.57:0/3202621065 shutdown_connections 2023-07-20T18:58:38.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.494+0000 7f86308eab80 1 -- 172.21.15.57:0/3202621065 wait complete. 2023-07-20T18:58:38.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.494+0000 7f86308eab80 1 -- start start 2023-07-20T18:58:38.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.494+0000 7f86308eab80 1 Processor -- start 2023-07-20T18:58:38.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f86308eab80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1ada540 0x55b6a1af2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f86308eab80 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1af2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f86308eab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adb900 0x55b6a1b42e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f86308eab80 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a1af8010 con 0x55b6a1adaea0 2023-07-20T18:58:38.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f86308eab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a1af8180 con 0x55b6a1adb900 2023-07-20T18:58:38.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f86308eab80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a1af8340 con 0x55b6a1ada540 2023-07-20T18:58:38.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f86308eab80 10 client.0 _ll_register_callbacks cb 0x55b6a1ad5890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-20T18:58:38.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f8615d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1af2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f8615d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1af2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44840/0 (socket says 172.21.15.57:44840) 2023-07-20T18:58:38.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f8615d9b700 1 -- 172.21.15.57:0/3528493606 learned_addr learned my addr 172.21.15.57:0/3528493606 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:38.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f8616d9d700 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adb900 0x55b6a1b42e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f861659c700 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1ada540 0x55b6a1af2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f8615d9b700 1 -- 172.21.15.57:0/3528493606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1ada540 msgr2=0x55b6a1af2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T18:58:38.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f8615d9b700 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1ada540 0x55b6a1af2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f8615d9b700 1 -- 172.21.15.57:0/3528493606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adb900 msgr2=0x55b6a1b42e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T18:58:38.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f8615d9b700 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adb900 0x55b6a1b42e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f8615d9b700 1 -- 172.21.15.57:0/3528493606 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861810b6c0 con 0x55b6a1adaea0 2023-07-20T18:58:38.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f8615d9b700 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1af2fe0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f860c096bc0 tx=0x7f860c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ab886cc267456a server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:38.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.495+0000 7f85fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-07-20T18:58:38.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f85fdffb700 1 -- 172.21.15.57:0/3528493606 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c0a25a0 con 0x55b6a1adaea0 2023-07-20T18:58:38.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f85fdffb700 1 -- 172.21.15.57:0/3528493606 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f860c09a150 con 0x55b6a1adaea0 2023-07-20T18:58:38.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f86308eab80 1 -- 172.21.15.57:0/3528493606 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f860c01e2e0 con 0x55b6a1adaea0 2023-07-20T18:58:38.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f85fdffb700 1 -- 172.21.15.57:0/3528493606 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f860c09e180 con 0x55b6a1adaea0 2023-07-20T18:58:38.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f86308eab80 10 client.14634 fetch_fsmap learned FSMap version 17 2023-07-20T18:58:38.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f86308eab80 1 -- 172.21.15.57:0/3528493606 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55b6a187f200 con 0x55b6a1adaea0 2023-07-20T18:58:38.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f85fdffb700 1 -- 172.21.15.57:0/3528493606 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f860c01e2e0 con 0x55b6a1adaea0 2023-07-20T18:58:38.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f85fdffb700 1 -- 172.21.15.57:0/3528493606 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f860c0a2210 con 0x55b6a1adaea0 2023-07-20T18:58:38.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f86308eab80 10 client.14634 fetch_fsmap finished waiting for FSMap version 17 2023-07-20T18:58:38.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f86308eab80 10 client.14634 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-20T18:58:38.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f86308eab80 20 client.14634 populate_metadata read hostname 'smithi057' 2023-07-20T18:58:38.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f86308eab80 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] conn(0x55b6a1b47b00 0x55b6a1b44460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:38.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f86308eab80 4 client.14634 mds_command: new command op to 24566 tid=0[{"prefix": "status"}] 2023-07-20T18:58:38.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.496+0000 7f86308eab80 1 -- 172.21.15.57:0/3528493606 --> [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b6a1a24730 con 0x55b6a1b47b00 2023-07-20T18:58:38.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.497+0000 7f861659c700 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] conn(0x55b6a1b47b00 0x55b6a1b44460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:38.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.497+0000 7f85fdffb700 10 client.14634 ms_handle_connect on v2:172.21.15.88:6838/2098157844 2023-07-20T18:58:38.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.497+0000 7f861659c700 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] conn(0x55b6a1b47b00 0x55b6a1b44460 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=565eb204ec7963a3 server_cookie=a00d51f914cece57 in_seq=0 out_seq=0 2023-07-20T18:58:38.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.497+0000 7f85fdffb700 1 -- 172.21.15.57:0/3528493606 <== mds.0 v2:172.21.15.88:6838/2098157844 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55b6a1a24730 con 0x55b6a1b47b00 2023-07-20T18:58:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.497+0000 7f85fdffb700 10 client.14634 handle_command_reply: tid=0 2023-07-20T18:58:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.497+0000 7f86308eab80 1 client.14634 shutdown 2023-07-20T18:58:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.497+0000 7f86308eab80 1 -- 172.21.15.57:0/3528493606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 msgr2=0x55b6a1af2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.497+0000 7f86308eab80 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1af2fe0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f860c096bc0 tx=0x7f860c009f60 comp rx=0 tx=0).stop 2023-07-20T18:58:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f86308eab80 1 -- 172.21.15.57:0/3528493606 shutdown_connections 2023-07-20T18:58:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f86308eab80 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.88:6838/2098157844,v1:172.21.15.88:6839/2098157844] conn(0x55b6a1b47b00 0x55b6a1b44460 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f86308eab80 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6a1ada540 0x55b6a1af2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f86308eab80 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6a1adb900 0x55b6a1b42e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f86308eab80 1 --2- 172.21.15.57:0/3528493606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b6a1adaea0 0x55b6a1af2fe0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f85fdffb700 0 client.14634 ms_handle_reset on v2:172.21.15.88:6838/2098157844 2023-07-20T18:58:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f86308eab80 1 -- 172.21.15.57:0/3528493606 >> 172.21.15.57:0/3528493606 conn(0x55b6a1a37c80 msgr2=0x55b6a1aee620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f86308eab80 1 -- 172.21.15.57:0/3528493606 shutdown_connections 2023-07-20T18:58:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f86308eab80 1 -- 172.21.15.57:0/3528493606 wait complete. 2023-07-20T18:58:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f86308eab80 20 client.14634 trim_cache size 0 max 16384 2023-07-20T18:58:38.561 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-07-20T18:58:38.561 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "38a58c3c-272e-11ee-9b34-001a4aab830c", 2023-07-20T18:58:38.561 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-07-20T18:58:38.561 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 24566, 2023-07-20T18:58:38.561 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-07-20T18:58:38.561 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-07-20T18:58:38.562 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-07-20T18:58:38.562 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 7.6600079919999997, 2023-07-20T18:58:38.562 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 17, 2023-07-20T18:58:38.562 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 86, 2023-07-20T18:58:38.562 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 85, 2023-07-20T18:58:38.562 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 13.197577914 2023-07-20T18:58:38.562 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-07-20T18:58:38.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f861cfff700 1 -- 172.21.15.57:0/3444065558 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f86000a1150 msgr2=0x7f86000a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f861cfff700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f86000a1150 0x7f86000a3610 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f861000f590 tx=0x7f861000b040 comp rx=0 tx=0).stop 2023-07-20T18:58:38.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f861cfff700 1 -- 172.21.15.57:0/3444065558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8618074750 msgr2=0x7f8618198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:38.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f861cfff700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8618074750 0x7f8618198c60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f860c00eb30 tx=0x7f860c00ee40 comp rx=0 tx=0).stop 2023-07-20T18:58:38.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f861cfff700 1 -- 172.21.15.57:0/3444065558 shutdown_connections 2023-07-20T18:58:38.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.498+0000 7f861cfff700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f86000a1150 0x7f86000a3610 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.499+0000 7f861cfff700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8618108630 0x7f861819fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.499+0000 7f861cfff700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8618073170 0x7f8618198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.499+0000 7f861cfff700 1 --2- 172.21.15.57:0/3444065558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8618074750 0x7f8618198c60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:38.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.499+0000 7f861cfff700 1 -- 172.21.15.57:0/3444065558 >> 172.21.15.57:0/3444065558 conn(0x7f86180fbcb0 msgr2=0x7f86180fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:38.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.499+0000 7f861cfff700 1 -- 172.21.15.57:0/3444065558 shutdown_connections 2023-07-20T18:58:38.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:38.499+0000 7f861cfff700 1 -- 172.21.15.57:0/3444065558 wait complete. 2023-07-20T18:58:38.566 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-20T18:58:38.567 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-20T18:58:38.567 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c -- ceph fs ls 2023-07-20T18:58:39.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:38 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1477724963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:39.293 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T18:58:39.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:38 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1477724963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:39.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:38 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1477724963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:40.126 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:39 smithi057 ceph-mon[126340]: pgmap v274: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 5 op/s 2023-07-20T18:58:40.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:39 smithi192 ceph-mon[129200]: pgmap v274: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 5 op/s 2023-07-20T18:58:40.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:39 smithi088 ceph-mon[128823]: pgmap v274: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 5 op/s 2023-07-20T18:58:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.739+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/1475828513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4c8100790 msgr2=0x7ff4c8100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:40.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.739+0000 7ff4cdf8d700 1 --2- 172.21.15.57:0/1475828513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4c8100790 0x7ff4c8100c10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009990 tx=0x7ff4bc009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:40.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.739+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/1475828513 shutdown_connections 2023-07-20T18:58:40.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.739+0000 7ff4cdf8d700 1 --2- 172.21.15.57:0/1475828513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8101150 0x7ff4c810d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:40.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.739+0000 7ff4cdf8d700 1 --2- 172.21.15.57:0/1475828513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c8106790 0x7ff4c8106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:40.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.739+0000 7ff4cdf8d700 1 --2- 172.21.15.57:0/1475828513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4c8100790 0x7ff4c8100c10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:40.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.739+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/1475828513 >> 172.21.15.57:0/1475828513 conn(0x7ff4c80fc260 msgr2=0x7ff4c80fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:40.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.739+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/1475828513 shutdown_connections 2023-07-20T18:58:40.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.740+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/1475828513 wait complete. 2023-07-20T18:58:40.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.741+0000 7ff4cdf8d700 1 Processor -- start 2023-07-20T18:58:40.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.741+0000 7ff4cdf8d700 1 -- start start 2023-07-20T18:58:40.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.741+0000 7ff4cdf8d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c8100790 0x7ff4c819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:40.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.741+0000 7ff4cdf8d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4c8101150 0x7ff4c819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:40.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.741+0000 7ff4cdf8d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8106790 0x7ff4c81989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:40.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.741+0000 7ff4cdf8d700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c8199190 con 0x7ff4c8101150 2023-07-20T18:58:40.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.741+0000 7ff4cdf8d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c8199300 con 0x7ff4c8100790 2023-07-20T18:58:40.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.741+0000 7ff4cdf8d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81994c0 con 0x7ff4c8106790 2023-07-20T18:58:40.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c77fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c8100790 0x7ff4c819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:40.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c6ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4c8101150 0x7ff4c819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:40.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c6ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4c8101150 0x7ff4c819ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44860/0 (socket says 172.21.15.57:44860) 2023-07-20T18:58:40.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c77fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c8100790 0x7ff4c819e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:58364/0 (socket says 172.21.15.57:58364) 2023-07-20T18:58:40.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c6ffd700 1 -- 172.21.15.57:0/3808800853 learned_addr learned my addr 172.21.15.57:0/3808800853 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:40.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c7fff700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8106790 0x7ff4c81989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:40.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c6ffd700 1 -- 172.21.15.57:0/3808800853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8106790 msgr2=0x7ff4c81989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:40.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c6ffd700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8106790 0x7ff4c81989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:40.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c6ffd700 1 -- 172.21.15.57:0/3808800853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c8100790 msgr2=0x7ff4c819e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:40.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c6ffd700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c8100790 0x7ff4c819e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:40.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c6ffd700 1 -- 172.21.15.57:0/3808800853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc0095f0 con 0x7ff4c8101150 2023-07-20T18:58:40.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c77fe700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c8100790 0x7ff4c819e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:40.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.742+0000 7ff4c6ffd700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4c8101150 0x7ff4c819ef30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc005910 tx=0x7ff4bc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:40.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.743+0000 7ff4c4ff9700 1 -- 172.21.15.57:0/3808800853 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc01d070 con 0x7ff4c8101150 2023-07-20T18:58:40.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.743+0000 7ff4c4ff9700 1 -- 172.21.15.57:0/3808800853 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7ff4bc00fb40 con 0x7ff4c8101150 2023-07-20T18:58:40.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.743+0000 7ff4c4ff9700 1 -- 172.21.15.57:0/3808800853 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0173f0 con 0x7ff4c8101150 2023-07-20T18:58:40.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.743+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/3808800853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c8199740 con 0x7ff4c8101150 2023-07-20T18:58:40.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.743+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/3808800853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c81a5970 con 0x7ff4c8101150 2023-07-20T18:58:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.744+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/3808800853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c804e470 con 0x7ff4c8101150 2023-07-20T18:58:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.746+0000 7ff4c4ff9700 1 -- 172.21.15.57:0/3808800853 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff4bc00fcb0 con 0x7ff4c8101150 2023-07-20T18:58:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.747+0000 7ff4c4ff9700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff4b00a11d0 0x7ff4b00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.747+0000 7ff4c4ff9700 1 -- 172.21.15.57:0/3808800853 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff4bc0ce1b0 con 0x7ff4c8101150 2023-07-20T18:58:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.748+0000 7ff4c77fe700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff4b00a11d0 0x7ff4b00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.748+0000 7ff4c77fe700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff4b00a11d0 0x7ff4b00a3690 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8009fd0 tx=0x7ff4b8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:40.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:40.750+0000 7ff4c4ff9700 1 -- 172.21.15.57:0/3808800853 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff4bc097740 con 0x7ff4c8101150 2023-07-20T18:58:41.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.003+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/3808800853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff4c81a5ba0 con 0x7ff4c8101150 2023-07-20T18:58:41.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.004+0000 7ff4c4ff9700 1 -- 172.21.15.57:0/3808800853 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff4bc031070 con 0x7ff4c8101150 2023-07-20T18:58:41.054 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-07-20T18:58:41.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.006+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/3808800853 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff4b00a11d0 msgr2=0x7ff4b00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:41.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.006+0000 7ff4cdf8d700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff4b00a11d0 0x7ff4b00a3690 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8009fd0 tx=0x7ff4b8009260 comp rx=0 tx=0).stop 2023-07-20T18:58:41.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.006+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/3808800853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4c8101150 msgr2=0x7ff4c819ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:41.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.006+0000 7ff4cdf8d700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4c8101150 0x7ff4c819ef30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc005910 tx=0x7ff4bc005740 comp rx=0 tx=0).stop 2023-07-20T18:58:41.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.006+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/3808800853 shutdown_connections 2023-07-20T18:58:41.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.006+0000 7ff4cdf8d700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7ff4b00a11d0 0x7ff4b00a3690 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:41.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.006+0000 7ff4cdf8d700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8106790 0x7ff4c81989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:41.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.006+0000 7ff4cdf8d700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c8100790 0x7ff4c819e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:41.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.006+0000 7ff4cdf8d700 1 --2- 172.21.15.57:0/3808800853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4c8101150 0x7ff4c819ef30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:41.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.006+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/3808800853 >> 172.21.15.57:0/3808800853 conn(0x7ff4c80fc260 msgr2=0x7ff4c810bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:41.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.007+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/3808800853 shutdown_connections 2023-07-20T18:58:41.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:41.007+0000 7ff4cdf8d700 1 -- 172.21.15.57:0/3808800853 wait complete. 2023-07-20T18:58:41.472 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-07-20T18:58:41.472 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-20T18:58:41.473 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-07-20T18:58:41.473 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-07-20T18:58:41.473 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-20T18:58:41.473 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-07-20T18:58:41.473 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-20T18:58:41.474 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-20T18:58:41.474 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-07-20T18:58:41.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:41.475 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-07-20T18:58:41.519 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-20T18:58:41.520 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-20T18:58:41.520 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-07-20T18:58:41.520 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-07-20T18:58:41.520 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-07-20T18:58:41.520 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-07-20T18:58:41.521 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-20T18:58:41.521 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-07-20T18:58:41.521 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-07-20T18:58:41.521 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-20T18:58:41.521 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-07-20T18:58:41.521 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-07-20T18:58:41.521 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-20T18:58:41.521 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-07-20T18:58:41.522 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-07-20T18:58:41.522 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-07-20T18:58:41.522 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 41543sec preferred_lft 41543sec 2023-07-20T18:58:41.522 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link 2023-07-20T18:58:41.522 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-07-20T18:58:41.522 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-20T18:58:41.522 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-07-20T18:58:41.522 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-07-20T18:58:41.524 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-20T18:58:41.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:41.524 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-07-20T18:58:41.525 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-07-20T18:58:41.525 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add name ceph-brx type bridge 2023-07-20T18:58:41.525 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr flush dev ceph-brx 2023-07-20T18:58:41.525 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set ceph-brx up 2023-07-20T18:58:41.526 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-20T18:58:41.526 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-07-20T18:58:42.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:42.003 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-20T18:58:42.077 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-07-20T18:58:42.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:42.081 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-07-20T18:58:42.149 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-07-20T18:58:42.149 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-20T18:58:42.149 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-07-20T18:58:42.150 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-07-20T18:58:42.150 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-20T18:58:42.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:42.151 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-07-20T18:58:42.151 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-07-20T18:58:42.152 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-07-20T18:58:42.152 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-07-20T18:58:42.152 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-07-20T18:58:42.153 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-07-20T18:58:42.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:41 smithi057 ceph-mon[126340]: pgmap v275: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 598 B/s wr, 1 op/s 2023-07-20T18:58:42.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:41 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3808800853' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-20T18:58:42.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:41 smithi192 ceph-mon[129200]: pgmap v275: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 598 B/s wr, 1 op/s 2023-07-20T18:58:42.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:41 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3808800853' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-20T18:58:42.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:41 smithi088 ceph-mon[128823]: pgmap v275: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 598 B/s wr, 1 op/s 2023-07-20T18:58:42.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:41 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3808800853' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-20T18:58:42.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:42.878 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-07-20T18:58:42.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:42.923 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list-id 2023-07-20T18:58:43.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:43.007 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-07-20T18:58:43.007 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-07-20T18:58:43.008 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-07-20T18:58:43.008 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-07-20T18:58:43.008 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-07-20T18:58:43.196 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-07-20T18:58:43.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:43.197 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-07-20T18:58:43.197 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-07-20T18:58:43.197 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-07-20T18:58:43.198 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-20T18:58:43.198 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-07-20T18:58:43.198 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-07-20T18:58:43.199 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-07-20T18:58:43.199 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-07-20T18:58:43.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:43.475 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-07-20T18:58:43.475 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-07-20T18:58:43.475 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-07-20T18:58:43.476 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set dev brx.0 master ceph-brx 2023-07-20T18:58:43.476 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-07-20T18:58:43.651 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-07-20T18:58:43.652 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-20T18:58:43.653 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-07-20T18:58:43.697 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-07-20T18:58:43.697 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-20T18:58:43.697 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-07-20T18:58:43.780 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-07-20T18:58:43.890 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-20T18:58:43.891 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-20T18:58:43.961 INFO:teuthology.orchestra.run.smithi057.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-20T18:58:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-20T18:58:43.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:43.965 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-07-20T18:58:44.009 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-07-20T18:58:44.009 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-07-20T18:58:44.054 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-07-20T18:58:44.127 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-20T18:58:44.127 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-20T18:58:44.158 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:43 smithi057 ceph-mon[126340]: pgmap v276: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-20T18:58:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-20T18:58:44.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:44.201 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-07-20T18:58:44.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:2023-07-20T18:58:44.212+0000 7f60920b95c0 -1 init, newargv = 0x55ea6ce99860 newargc=15 2023-07-20T18:58:44.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:ceph-fuse[163650]: starting ceph client 2023-07-20T18:58:44.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:ceph-fuse[163650]: starting fuse 2023-07-20T18:58:44.292 INFO:teuthology.orchestra.run.smithi057.stdout:121 2023-07-20T18:58:44.292 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [121] 2023-07-20T18:58:44.293 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-07-20T18:58:44.293 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-07-20T18:58:44.294 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-07-20T18:58:44.295 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-07-20T18:58:44.295 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-07-20T18:58:44.296 DEBUG:teuthology.orchestra.run.smithi057:> 2023-07-20T18:58:44.296 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-07-20T18:58:44.297 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-07-20T18:58:44.298 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-07-20T18:58:44.298 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-07-20T18:58:44.299 DEBUG:teuthology.orchestra.run.smithi057:> 2023-07-20T18:58:44.300 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-07-20T18:58:44.300 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-07-20T18:58:44.301 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-07-20T18:58:44.301 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-07-20T18:58:44.301 DEBUG:teuthology.orchestra.run.smithi057:> 2023-07-20T18:58:44.302 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-07-20T18:58:44.302 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-07-20T18:58:44.302 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-07-20T18:58:44.302 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-07-20T18:58:44.303 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-07-20T18:58:44.303 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-07-20T18:58:44.303 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-07-20T18:58:44.303 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-07-20T18:58:44.304 DEBUG:teuthology.orchestra.run.smithi057:> 2023-07-20T18:58:44.304 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.admin")) 2023-07-20T18:58:44.304 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-07-20T18:58:44.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:43 smithi192 ceph-mon[129200]: pgmap v276: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:44.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:43 smithi088 ceph-mon[128823]: pgmap v276: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:44.479 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.admin.163650.asok 2023-07-20T18:58:44.657 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.163650.asok 2023-07-20T18:58:44.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:44.658 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.163650.asok status 2023-07-20T18:58:44.891 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-07-20T18:58:44.891 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-07-20T18:58:44.891 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "1b0b2eb005a7c51d97a788b9900d511e1394b537", 2023-07-20T18:58:44.891 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T18:58:44.891 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "admin", 2023-07-20T18:58:44.891 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-07-20T18:58:44.892 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-07-20T18:58:44.892 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "163650", 2023-07-20T18:58:44.892 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-07-20T18:58:44.892 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-07-20T18:58:44.892 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-07-20T18:58:44.892 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-07-20T18:58:44.893 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 24691, 2023-07-20T18:58:44.893 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-07-20T18:58:44.893 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-07-20T18:58:44.893 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-07-20T18:58:44.893 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 24691 2023-07-20T18:58:44.893 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-07-20T18:58:44.893 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-07-20T18:58:44.894 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-07-20T18:58:44.894 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:0", 2023-07-20T18:58:44.894 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 1665488073 2023-07-20T18:58:44.894 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-07-20T18:58:44.894 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-07-20T18:58:44.894 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-07-20T18:58:44.895 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-07-20T18:58:44.895 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:0", 2023-07-20T18:58:44.895 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 1665488073 2023-07-20T18:58:44.895 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-07-20T18:58:44.895 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.24691 172.21.15.57:0/1665488073", 2023-07-20T18:58:44.895 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "172.21.15.57:0/1665488073", 2023-07-20T18:58:44.895 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-07-20T18:58:44.895 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 17, 2023-07-20T18:58:44.896 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 86, 2023-07-20T18:58:44.896 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-07-20T18:58:44.896 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-07-20T18:58:44.896 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-07-20T18:58:44.896 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-07-20T18:58:44.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:44.906 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-07-20T18:58:44.906 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-07-20T18:58:44.951 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-07-20T18:58:44.951 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-07-20T18:58:44.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:44.952 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-07-20T18:58:45.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:45.064 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-07-20T18:58:45.193 INFO:teuthology.orchestra.run.smithi057.stdout:# file: . 2023-07-20T18:58:45.194 INFO:teuthology.orchestra.run.smithi057.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-07-20T18:58:45.194 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:45.194 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:58:45.195 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-20T18:58:45.241 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T18:58:45.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.436+0000 7f54c2dff700 1 -- 172.21.15.57:0/1801928269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54bc102300 msgr2=0x7f54bc108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.436+0000 7f54c2dff700 1 --2- 172.21.15.57:0/1801928269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54bc102300 0x7f54bc108bb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f54b400b0d0 tx=0x7f54b400b3e0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.436+0000 7f54c2dff700 1 -- 172.21.15.57:0/1801928269 shutdown_connections 2023-07-20T18:58:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.436+0000 7f54c2dff700 1 --2- 172.21.15.57:0/1801928269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54bc100770 0x7f54bc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.436+0000 7f54c2dff700 1 --2- 172.21.15.57:0/1801928269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54bc102300 0x7f54bc108bb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.436+0000 7f54c2dff700 1 --2- 172.21.15.57:0/1801928269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54bc101960 0x7f54bc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.436+0000 7f54c2dff700 1 -- 172.21.15.57:0/1801928269 >> 172.21.15.57:0/1801928269 conn(0x7f54bc0fbca0 msgr2=0x7f54bc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.436+0000 7f54c2dff700 1 -- 172.21.15.57:0/1801928269 shutdown_connections 2023-07-20T18:58:45.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.436+0000 7f54c2dff700 1 -- 172.21.15.57:0/1801928269 wait complete. 2023-07-20T18:58:45.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.437+0000 7f54c2dff700 1 Processor -- start 2023-07-20T18:58:45.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.437+0000 7f54c2dff700 1 -- start start 2023-07-20T18:58:45.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.437+0000 7f54c2dff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54bc100770 0x7f54bc074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54c2dff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54bc101960 0x7f54bc072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54c2dff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54bc102300 0x7f54bc0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54c2dff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc073840 con 0x7f54bc101960 2023-07-20T18:58:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54c2dff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0739b0 con 0x7f54bc100770 2023-07-20T18:58:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54c2dff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc1a0660 con 0x7f54bc102300 2023-07-20T18:58:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54bbfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54bc101960 0x7f54bc072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54c0b9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54bc100770 0x7f54bc074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54bbfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54bc101960 0x7f54bc072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54772/0 (socket says 172.21.15.57:54772) 2023-07-20T18:58:45.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54c0b9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54bc100770 0x7f54bc074520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:38094/0 (socket says 172.21.15.57:38094) 2023-07-20T18:58:45.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54bbfff700 1 -- 172.21.15.57:0/1629667058 learned_addr learned my addr 172.21.15.57:0/1629667058 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:45.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54c139c700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54bc102300 0x7f54bc0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:45.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54bbfff700 1 -- 172.21.15.57:0/1629667058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54bc102300 msgr2=0x7f54bc0730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:45.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54bbfff700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54bc102300 0x7f54bc0730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54bbfff700 1 -- 172.21.15.57:0/1629667058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54bc100770 msgr2=0x7f54bc074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:45.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54bbfff700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54bc100770 0x7f54bc074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54bbfff700 1 -- 172.21.15.57:0/1629667058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b4009d00 con 0x7f54bc101960 2023-07-20T18:58:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.438+0000 7f54c139c700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54bc102300 0x7f54bc0730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T18:58:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.439+0000 7f54bbfff700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54bc101960 0x7f54bc072b60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f54a800e9a0 tx=0x7f54a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.439+0000 7f54b9ffb700 1 -- 172.21.15.57:0/1629667058 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8009790 con 0x7f54bc101960 2023-07-20T18:58:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.439+0000 7f54b9ffb700 1 -- 172.21.15.57:0/1629667058 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f54a800cbd0 con 0x7f54bc101960 2023-07-20T18:58:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.439+0000 7f54b9ffb700 1 -- 172.21.15.57:0/1629667058 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8018680 con 0x7f54bc101960 2023-07-20T18:58:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.439+0000 7f54c2dff700 1 -- 172.21.15.57:0/1629667058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54bc1a0940 con 0x7f54bc101960 2023-07-20T18:58:45.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.439+0000 7f54c2dff700 1 -- 172.21.15.57:0/1629667058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc1a0e50 con 0x7f54bc101960 2023-07-20T18:58:45.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.440+0000 7f54c2dff700 1 -- 172.21.15.57:0/1629667058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54bc107c10 con 0x7f54bc101960 2023-07-20T18:58:45.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.442+0000 7f54b9ffb700 1 -- 172.21.15.57:0/1629667058 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f54a80188a0 con 0x7f54bc101960 2023-07-20T18:58:45.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.443+0000 7f54b9ffb700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f54ac0a1220 0x7f54ac0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:45.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.443+0000 7f54b9ffb700 1 -- 172.21.15.57:0/1629667058 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f54a8014070 con 0x7f54bc101960 2023-07-20T18:58:45.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.444+0000 7f54c0b9b700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f54ac0a1220 0x7f54ac0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:45.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.444+0000 7f54c0b9b700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f54ac0a1220 0x7f54ac0a36e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f54bc1017c0 tx=0x7f54b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:45.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.448+0000 7f54b9ffb700 1 -- 172.21.15.57:0/1629667058 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f54a8051780 con 0x7f54bc101960 2023-07-20T18:58:45.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.657+0000 7f54c2dff700 1 -- 172.21.15.57:0/1629667058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54bc02c970 con 0x7f54bc101960 2023-07-20T18:58:45.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.659+0000 7f54b9ffb700 1 -- 172.21.15.57:0/1629667058 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23911 (secure 0 0 0) 0x7f54a80962f0 con 0x7f54bc101960 2023-07-20T18:58:45.709 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:45.710 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":86,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","created":"2023-07-20T18:51:01.245641+0000","modified":"2023-07-20T18:58:35.383850+0000","last_up_change":"2023-07-20T18:57:44.823544+0000","last_in_change":"2023-07-20T18:57:27.428782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T18:54:23.215777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T18:58:28.447992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T18:58:29.772740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T18:58:33.395093+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"627a67ea-1366-4b3f-babd-067f75bad1e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6803","nonce":3023441607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6805","nonce":3023441607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6809","nonce":3023441607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6807","nonce":3023441607}]},"public_addr":"172.21.15.57:6803/3023441607","cluster_addr":"172.21.15.57:6805/3023441607","heartbeat_back_addr":"172.21.15.57:6809/3023441607","heartbeat_front_addr":"172.21.15.57:6807/3023441607","state":["exists","up"]},{"osd":1,"uuid":"adcdb236-334e-4478-98a8-7d0628492082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6801","nonce":1809528555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6803","nonce":1809528555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6807","nonce":1809528555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6805","nonce":1809528555}]},"public_addr":"172.21.15.88:6801/1809528555","cluster_addr":"172.21.15.88:6803/1809528555","heartbeat_back_addr":"172.21.15.88:6807/1809528555","heartbeat_front_addr":"172.21.15.88:6805/1809528555","state":["exists","up"]},{"osd":2,"uuid":"3cd56229-2067-433a-b9f9-f47b1b878b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6801","nonce":3891687382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6803","nonce":3891687382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6807","nonce":3891687382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6805","nonce":3891687382}]},"public_addr":"172.21.15.192:6801/3891687382","cluster_addr":"172.21.15.192:6803/3891687382","heartbeat_back_addr":"172.21.15.192:6807/3891687382","heartbeat_front_addr":"172.21.15.192:6805/3891687382","state":["exists","up"]},{"osd":3,"uuid":"aae250ab-e242-4ef8-9c3a-a4fa36102c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6811","nonce":2920717314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6813","nonce":2920717314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6817","nonce":2920717314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6815","nonce":2920717314}]},"public_addr":"172.21.15.57:6811/2920717314","cluster_addr":"172.21.15.57:6813/2920717314","heartbeat_back_addr":"172.21.15.57:6817/2920717314","heartbeat_front_addr":"172.21.15.57:6815/2920717314","state":["exists","up"]},{"osd":4,"uuid":"8c8d1dfa-1cff-4f0a-ae72-22bd522ed696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6809","nonce":3911525610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6811","nonce":3911525610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6815","nonce":3911525610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6813","nonce":3911525610}]},"public_addr":"172.21.15.88:6809/3911525610","cluster_addr":"172.21.15.88:6811/3911525610","heartbeat_back_addr":"172.21.15.88:6815/3911525610","heartbeat_front_addr":"172.21.15.88:6813/3911525610","state":["exists","up"]},{"osd":5,"uuid":"5a2a56ab-2af9-4176-ba4d-b2138f0e7c75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6809","nonce":905436397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6811","nonce":905436397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6815","nonce":905436397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6813","nonce":905436397}]},"public_addr":"172.21.15.192:6809/905436397","cluster_addr":"172.21.15.192:6811/905436397","heartbeat_back_addr":"172.21.15.192:6815/905436397","heartbeat_front_addr":"172.21.15.192:6813/905436397","state":["exists","up"]},{"osd":6,"uuid":"03eafdda-278c-443c-b376-677eeb50262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6819","nonce":333894423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6821","nonce":333894423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6825","nonce":333894423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6823","nonce":333894423}]},"public_addr":"172.21.15.57:6819/333894423","cluster_addr":"172.21.15.57:6821/333894423","heartbeat_back_addr":"172.21.15.57:6825/333894423","heartbeat_front_addr":"172.21.15.57:6823/333894423","state":["exists","up"]},{"osd":7,"uuid":"3f01254e-be73-41c8-b979-3b6a7534e2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6817","nonce":3031485551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6819","nonce":3031485551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6823","nonce":3031485551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6821","nonce":3031485551}]},"public_addr":"172.21.15.88:6817/3031485551","cluster_addr":"172.21.15.88:6819/3031485551","heartbeat_back_addr":"172.21.15.88:6823/3031485551","heartbeat_front_addr":"172.21.15.88:6821/3031485551","state":["exists","up"]},{"osd":8,"uuid":"624a150e-f47b-4a76-8785-c818dd65342e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6817","nonce":918056390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6819","nonce":918056390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6823","nonce":918056390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6821","nonce":918056390}]},"public_addr":"172.21.15.192:6817/918056390","cluster_addr":"172.21.15.192:6819/918056390","heartbeat_back_addr":"172.21.15.192:6823/918056390","heartbeat_front_addr":"172.21.15.192:6821/918056390","state":["exists","up"]},{"osd":9,"uuid":"6d9cb69d-ed7b-4009-9ec5-027becc577d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6827","nonce":1773340763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6829","nonce":1773340763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6833","nonce":1773340763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6831","nonce":1773340763}]},"public_addr":"172.21.15.57:6827/1773340763","cluster_addr":"172.21.15.57:6829/1773340763","heartbeat_back_addr":"172.21.15.57:6833/1773340763","heartbeat_front_addr":"172.21.15.57:6831/1773340763","state":["exists","up"]},{"osd":10,"uuid":"d5799219-0979-40b1-a596-acd37f37a72d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6825","nonce":2781975579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6827","nonce":2781975579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6831","nonce":2781975579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6829","nonce":2781975579}]},"public_addr":"172.21.15.88:6825/2781975579","cluster_addr":"172.21.15.88:6827/2781975579","heartbeat_back_addr":"172.21.15.88:6831/2781975579","heartbeat_front_addr":"172.21.15.88:6829/2781975579","state":["exists","up"]},{"osd":11,"uuid":"e77cac6c-107e-41f9-8efd-e1140c4e3bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6825","nonce":2864190031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6827","nonce":2864190031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6831","nonce":2864190031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6829","nonce":2864190031}]},"public_addr":"172.21.15.192:6825/2864190031","cluster_addr":"172.21.15.192:6827/2864190031","heartbeat_back_addr":"172.21.15.192:6831/2864190031","heartbeat_front_addr":"172.21.15.192:6829/2864190031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:53:38.180971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:00.253039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:20.776122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:48.631094+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:09.288231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:29.595802+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:55.089126+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:15.637849+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:35.985101+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:01.544767+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:22.351743+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:43.255157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6801/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/3804015220":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6800/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1582727629":"2023-07-21T18:51:34.614904+0000","172.21.15.57:6800/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/2044626900":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/748511752":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1151100650":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6801/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/3446341201":"2023-07-21T18:52:02.610729+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T18:58:45.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.661+0000 7f54c2dff700 1 -- 172.21.15.57:0/1629667058 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f54ac0a1220 msgr2=0x7f54ac0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:45.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.661+0000 7f54c2dff700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f54ac0a1220 0x7f54ac0a36e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f54bc1017c0 tx=0x7f54b0009260 comp rx=0 tx=0).stop 2023-07-20T18:58:45.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.661+0000 7f54c2dff700 1 -- 172.21.15.57:0/1629667058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54bc101960 msgr2=0x7f54bc072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:45.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.661+0000 7f54c2dff700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54bc101960 0x7f54bc072b60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f54a800e9a0 tx=0x7f54a800ecb0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.661+0000 7f54c2dff700 1 -- 172.21.15.57:0/1629667058 shutdown_connections 2023-07-20T18:58:45.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.661+0000 7f54c2dff700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f54ac0a1220 0x7f54ac0a36e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.661+0000 7f54c2dff700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54bc102300 0x7f54bc0730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.661+0000 7f54c2dff700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54bc100770 0x7f54bc074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.661+0000 7f54c2dff700 1 --2- 172.21.15.57:0/1629667058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54bc101960 0x7f54bc072b60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.661+0000 7f54c2dff700 1 -- 172.21.15.57:0/1629667058 >> 172.21.15.57:0/1629667058 conn(0x7f54bc0fbca0 msgr2=0x7f54bc104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:45.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.662+0000 7f54c2dff700 1 -- 172.21.15.57:0/1629667058 shutdown_connections 2023-07-20T18:58:45.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.662+0000 7f54c2dff700 1 -- 172.21.15.57:0/1629667058 wait complete. 2023-07-20T18:58:45.729 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T18:54:23.215777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-20T18:58:28.447992+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-20T18:58:29.772740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-20T18:58:33.395093+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-20T18:58:45.730 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T18:58:45.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.890+0000 7fa2151ff700 1 -- 172.21.15.57:0/2485555706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa210073170 msgr2=0x7fa2100735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:45.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.890+0000 7fa2151ff700 1 --2- 172.21.15.57:0/2485555706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa210073170 0x7fa2100735f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa204009990 tx=0x7fa204009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.890+0000 7fa2151ff700 1 -- 172.21.15.57:0/2485555706 shutdown_connections 2023-07-20T18:58:45.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.890+0000 7fa2151ff700 1 --2- 172.21.15.57:0/2485555706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa210074750 0x7fa210072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.890+0000 7fa2151ff700 1 --2- 172.21.15.57:0/2485555706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210108630 0x7fa21010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.890+0000 7fa2151ff700 1 --2- 172.21.15.57:0/2485555706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa210073170 0x7fa2100735f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.890+0000 7fa2151ff700 1 -- 172.21.15.57:0/2485555706 >> 172.21.15.57:0/2485555706 conn(0x7fa2100fbcb0 msgr2=0x7fa2100fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:45.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.890+0000 7fa2151ff700 1 -- 172.21.15.57:0/2485555706 shutdown_connections 2023-07-20T18:58:45.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.890+0000 7fa2151ff700 1 -- 172.21.15.57:0/2485555706 wait complete. 2023-07-20T18:58:45.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.891+0000 7fa2151ff700 1 Processor -- start 2023-07-20T18:58:45.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.891+0000 7fa2151ff700 1 -- start start 2023-07-20T18:58:45.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa2151ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa210073170 0x7fa210198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:45.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa2151ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa210074750 0x7fa210198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:45.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa2151ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210108630 0x7fa21019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:45.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa2151ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2101a0290 con 0x7fa210073170 2023-07-20T18:58:45.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa2151ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2101a03d0 con 0x7fa210108630 2023-07-20T18:58:45.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa2151ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2101a0560 con 0x7fa210074750 2023-07-20T18:58:45.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa20ed9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa210073170 0x7fa210198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:45.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa20ed9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa210073170 0x7fa210198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54782/0 (socket says 172.21.15.57:54782) 2023-07-20T18:58:45.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa20ed9d700 1 -- 172.21.15.57:0/1046594995 learned_addr learned my addr 172.21.15.57:0/1046594995 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:45.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa20e59c700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa210074750 0x7fa210198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:45.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa20f59e700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210108630 0x7fa21019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:45.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa20ed9d700 1 -- 172.21.15.57:0/1046594995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa210074750 msgr2=0x7fa210198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:45.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa20ed9d700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa210074750 0x7fa210198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa20ed9d700 1 -- 172.21.15.57:0/1046594995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210108630 msgr2=0x7fa21019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:45.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa20ed9d700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210108630 0x7fa21019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:45.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.892+0000 7fa20ed9d700 1 -- 172.21.15.57:0/1046594995 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa208009c40 con 0x7fa210073170 2023-07-20T18:58:45.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.893+0000 7fa20ed9d700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa210073170 0x7fa210198710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa20000e9a0 tx=0x7fa20000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:45.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.893+0000 7fa1f7fff700 1 -- 172.21.15.57:0/1046594995 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200009790 con 0x7fa210073170 2023-07-20T18:58:45.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.893+0000 7fa2151ff700 1 -- 172.21.15.57:0/1046594995 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2040095f0 con 0x7fa210073170 2023-07-20T18:58:45.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.893+0000 7fa1f7fff700 1 -- 172.21.15.57:0/1046594995 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fa20000cbd0 con 0x7fa210073170 2023-07-20T18:58:45.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.893+0000 7fa1f7fff700 1 -- 172.21.15.57:0/1046594995 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200018730 con 0x7fa210073170 2023-07-20T18:58:45.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.893+0000 7fa2151ff700 1 -- 172.21.15.57:0/1046594995 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2101a0bc0 con 0x7fa210073170 2023-07-20T18:58:45.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.894+0000 7fa2151ff700 1 -- 172.21.15.57:0/1046594995 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa210106d30 con 0x7fa210073170 2023-07-20T18:58:45.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.896+0000 7fa1f7fff700 1 -- 172.21.15.57:0/1046594995 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa200010430 con 0x7fa210073170 2023-07-20T18:58:45.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.897+0000 7fa1f7fff700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa1f80a1150 0x7fa1f80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:45.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.898+0000 7fa1f7fff700 1 -- 172.21.15.57:0/1046594995 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa200014070 con 0x7fa210073170 2023-07-20T18:58:45.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.898+0000 7fa20e59c700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa1f80a1150 0x7fa1f80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:45.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.898+0000 7fa20e59c700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa1f80a1150 0x7fa1f80a3610 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa204006010 tx=0x7fa20400b260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:45.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:45.899+0000 7fa1f7fff700 1 -- 172.21.15.57:0/1046594995 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa200096840 con 0x7fa210073170 2023-07-20T18:58:45.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:45 smithi057 ceph-mon[126340]: pgmap v277: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:45.985 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:45 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1629667058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:46.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.110+0000 7fa2151ff700 1 -- 172.21.15.57:0/1046594995 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa21002cb20 con 0x7fa210073170 2023-07-20T18:58:46.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.110+0000 7fa1f7fff700 1 -- 172.21.15.57:0/1046594995 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa200099e60 con 0x7fa210073170 2023-07-20T18:58:46.160 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-07-20T18:58:46.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 -- 172.21.15.57:0/1046594995 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa1f80a1150 msgr2=0x7fa1f80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:46.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa1f80a1150 0x7fa1f80a3610 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa204006010 tx=0x7fa20400b260 comp rx=0 tx=0).stop 2023-07-20T18:58:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 -- 172.21.15.57:0/1046594995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa210073170 msgr2=0x7fa210198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa210073170 0x7fa210198710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa20000e9a0 tx=0x7fa20000ed60 comp rx=0 tx=0).stop 2023-07-20T18:58:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 -- 172.21.15.57:0/1046594995 shutdown_connections 2023-07-20T18:58:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fa1f80a1150 0x7fa1f80a3610 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210108630 0x7fa21019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa210073170 0x7fa210198710 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 --2- 172.21.15.57:0/1046594995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa210074750 0x7fa210198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 -- 172.21.15.57:0/1046594995 >> 172.21.15.57:0/1046594995 conn(0x7fa2100fbcb0 msgr2=0x7fa2100fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 -- 172.21.15.57:0/1046594995 shutdown_connections 2023-07-20T18:58:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.113+0000 7fa2151ff700 1 -- 172.21.15.57:0/1046594995 wait complete. 2023-07-20T18:58:46.181 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-20T18:58:46.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:45 smithi192 ceph-mon[129200]: pgmap v277: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:46.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:45 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1629667058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:46.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:45 smithi088 ceph-mon[128823]: pgmap v277: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:46.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:45 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1629667058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:46.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.340+0000 7faada7ff700 1 -- 172.21.15.57:0/1547824162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faad4073170 msgr2=0x7faad40735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:46.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.340+0000 7faada7ff700 1 --2- 172.21.15.57:0/1547824162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faad4073170 0x7faad40735f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faac80098f0 tx=0x7faac8009c00 comp rx=0 tx=0).stop 2023-07-20T18:58:46.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.340+0000 7faada7ff700 1 -- 172.21.15.57:0/1547824162 shutdown_connections 2023-07-20T18:58:46.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.340+0000 7faada7ff700 1 --2- 172.21.15.57:0/1547824162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faad4108630 0x7faad410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.340+0000 7faada7ff700 1 --2- 172.21.15.57:0/1547824162 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faad4074750 0x7faad4072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.340+0000 7faada7ff700 1 --2- 172.21.15.57:0/1547824162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faad4073170 0x7faad40735f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.340+0000 7faada7ff700 1 -- 172.21.15.57:0/1547824162 >> 172.21.15.57:0/1547824162 conn(0x7faad40fbcb0 msgr2=0x7faad40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:46.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.340+0000 7faada7ff700 1 -- 172.21.15.57:0/1547824162 shutdown_connections 2023-07-20T18:58:46.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.340+0000 7faada7ff700 1 -- 172.21.15.57:0/1547824162 wait complete. 2023-07-20T18:58:46.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.341+0000 7faada7ff700 1 Processor -- start 2023-07-20T18:58:46.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faada7ff700 1 -- start start 2023-07-20T18:58:46.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faada7ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faad4073170 0x7faad41987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:46.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faada7ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faad4074750 0x7faad4198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:46.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faada7ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faad4108630 0x7faad419fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:46.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faada7ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad41a0320 con 0x7faad4074750 2023-07-20T18:58:46.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faada7ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad41a0460 con 0x7faad4073170 2023-07-20T18:58:46.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faada7ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad41a05f0 con 0x7faad4108630 2023-07-20T18:58:46.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faad3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faad4073170 0x7faad41987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:46.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faad37fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faad4074750 0x7faad4198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:46.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faad3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faad4073170 0x7faad41987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:38150/0 (socket says 172.21.15.57:38150) 2023-07-20T18:58:46.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faad3fff700 1 -- 172.21.15.57:0/2925841973 learned_addr learned my addr 172.21.15.57:0/2925841973 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:46.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.342+0000 7faad8d9c700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faad4108630 0x7faad419fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:46.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faad3fff700 1 -- 172.21.15.57:0/2925841973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faad4108630 msgr2=0x7faad419fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:46.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faad3fff700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faad4108630 0x7faad419fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faad3fff700 1 -- 172.21.15.57:0/2925841973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faad4074750 msgr2=0x7faad4198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:46.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faad3fff700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faad4074750 0x7faad4198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faad3fff700 1 -- 172.21.15.57:0/2925841973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac8009520 con 0x7faad4073170 2023-07-20T18:58:46.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faad37fe700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faad4074750 0x7faad4198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:46.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faad3fff700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faad4073170 0x7faad41987a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faac400e9c0 tx=0x7faac400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:46.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faad17fa700 1 -- 172.21.15.57:0/2925841973 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4004a30 con 0x7faad4073170 2023-07-20T18:58:46.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faad17fa700 1 -- 172.21.15.57:0/2925841973 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7faac400cca0 con 0x7faad4073170 2023-07-20T18:58:46.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faada7ff700 1 -- 172.21.15.57:0/2925841973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad41a08d0 con 0x7faad4073170 2023-07-20T18:58:46.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faad17fa700 1 -- 172.21.15.57:0/2925841973 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4005580 con 0x7faad4073170 2023-07-20T18:58:46.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.343+0000 7faada7ff700 1 -- 172.21.15.57:0/2925841973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad41a0de0 con 0x7faad4073170 2023-07-20T18:58:46.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.344+0000 7faada7ff700 1 -- 172.21.15.57:0/2925841973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad4106d30 con 0x7faad4073170 2023-07-20T18:58:46.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.347+0000 7faad17fa700 1 -- 172.21.15.57:0/2925841973 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faac4004c20 con 0x7faad4073170 2023-07-20T18:58:46.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.348+0000 7faad17fa700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7faabc0a1150 0x7faabc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:46.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.348+0000 7faad17fa700 1 -- 172.21.15.57:0/2925841973 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7faac4014070 con 0x7faad4073170 2023-07-20T18:58:46.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.348+0000 7faad37fe700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7faabc0a1150 0x7faabc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:46.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.349+0000 7faad37fe700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7faabc0a1150 0x7faabc0a3610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faac8005e90 tx=0x7faac8019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:46.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.350+0000 7faad17fa700 1 -- 172.21.15.57:0/2925841973 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faac4096330 con 0x7faad4073170 2023-07-20T18:58:46.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.561+0000 7faada7ff700 1 -- 172.21.15.57:0/2925841973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faad402cbe0 con 0x7faad4073170 2023-07-20T18:58:46.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.562+0000 7faad17fa700 1 -- 172.21.15.57:0/2925841973 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7faac4099950 con 0x7faad4073170 2023-07-20T18:58:46.612 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-07-20T18:58:46.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 -- 172.21.15.57:0/2925841973 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7faabc0a1150 msgr2=0x7faabc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:46.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7faabc0a1150 0x7faabc0a3610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faac8005e90 tx=0x7faac8019040 comp rx=0 tx=0).stop 2023-07-20T18:58:46.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 -- 172.21.15.57:0/2925841973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faad4073170 msgr2=0x7faad41987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:46.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faad4073170 0x7faad41987a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faac400e9c0 tx=0x7faac400ecd0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 -- 172.21.15.57:0/2925841973 shutdown_connections 2023-07-20T18:58:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7faabc0a1150 0x7faabc0a3610 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faad4108630 0x7faad419fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faad4073170 0x7faad41987a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 --2- 172.21.15.57:0/2925841973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faad4074750 0x7faad4198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 -- 172.21.15.57:0/2925841973 >> 172.21.15.57:0/2925841973 conn(0x7faad40fbcb0 msgr2=0x7faad40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 -- 172.21.15.57:0/2925841973 shutdown_connections 2023-07-20T18:58:46.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.564+0000 7faada7ff700 1 -- 172.21.15.57:0/2925841973 wait complete. 2023-07-20T18:58:46.632 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-20T18:58:46.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.790+0000 7f41075ff700 1 -- 172.21.15.57:0/2619031258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4100073170 msgr2=0x7f41000735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:46.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.790+0000 7f41075ff700 1 --2- 172.21.15.57:0/2619031258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4100073170 0x7f41000735f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f40f00098f0 tx=0x7f40f0009c00 comp rx=0 tx=0).stop 2023-07-20T18:58:46.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.790+0000 7f41075ff700 1 -- 172.21.15.57:0/2619031258 shutdown_connections 2023-07-20T18:58:46.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.790+0000 7f41075ff700 1 --2- 172.21.15.57:0/2619031258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4100108630 0x7f410010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.790+0000 7f41075ff700 1 --2- 172.21.15.57:0/2619031258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4100074750 0x7f4100072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.790+0000 7f41075ff700 1 --2- 172.21.15.57:0/2619031258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4100073170 0x7f41000735f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.790+0000 7f41075ff700 1 -- 172.21.15.57:0/2619031258 >> 172.21.15.57:0/2619031258 conn(0x7f41000fbcb0 msgr2=0x7f41000fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:46.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.791+0000 7f41075ff700 1 -- 172.21.15.57:0/2619031258 shutdown_connections 2023-07-20T18:58:46.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.791+0000 7f41075ff700 1 -- 172.21.15.57:0/2619031258 wait complete. 2023-07-20T18:58:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.791+0000 7f41075ff700 1 Processor -- start 2023-07-20T18:58:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f41075ff700 1 -- start start 2023-07-20T18:58:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f41075ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4100073170 0x7f4100198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f41075ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4100074750 0x7f4100198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f41075ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4100108630 0x7f410019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f41075ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41001a02d0 con 0x7f4100074750 2023-07-20T18:58:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f41075ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41001a0410 con 0x7f4100108630 2023-07-20T18:58:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f41075ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41001a05a0 con 0x7f4100073170 2023-07-20T18:58:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f4104b9a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4100074750 0x7f4100198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f4104b9a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4100074750 0x7f4100198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54838/0 (socket says 172.21.15.57:54838) 2023-07-20T18:58:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f4104b9a700 1 -- 172.21.15.57:0/538408384 learned_addr learned my addr 172.21.15.57:0/538408384 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:46.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f410539b700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4100073170 0x7f4100198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:46.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.792+0000 7f4105b9c700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4100108630 0x7f410019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:46.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f4104b9a700 1 -- 172.21.15.57:0/538408384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4100073170 msgr2=0x7f4100198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:46.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f4104b9a700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4100073170 0x7f4100198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f4104b9a700 1 -- 172.21.15.57:0/538408384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4100108630 msgr2=0x7f410019fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:46.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f4104b9a700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4100108630 0x7f410019fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:46.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f4104b9a700 1 -- 172.21.15.57:0/538408384 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f8009d00 con 0x7f4100074750 2023-07-20T18:58:46.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f4104b9a700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4100074750 0x7f4100198c90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f40f0005270 tx=0x7f40f00052a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:46.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f40f67fc700 1 -- 172.21.15.57:0/538408384 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f001c070 con 0x7f4100074750 2023-07-20T18:58:46.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f41075ff700 1 -- 172.21.15.57:0/538408384 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f0009520 con 0x7f4100074750 2023-07-20T18:58:46.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f40f67fc700 1 -- 172.21.15.57:0/538408384 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f40f000f9a0 con 0x7f4100074750 2023-07-20T18:58:46.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f40f67fc700 1 -- 172.21.15.57:0/538408384 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0004a90 con 0x7f4100074750 2023-07-20T18:58:46.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.793+0000 7f41075ff700 1 -- 172.21.15.57:0/538408384 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41001a0ab0 con 0x7f4100074750 2023-07-20T18:58:46.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.794+0000 7f41075ff700 1 -- 172.21.15.57:0/538408384 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4100106d30 con 0x7f4100074750 2023-07-20T18:58:46.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.796+0000 7f40f67fc700 1 -- 172.21.15.57:0/538408384 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f40f0020070 con 0x7f4100074750 2023-07-20T18:58:46.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.797+0000 7f40f67fc700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40ec0a1150 0x7f40ec0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:46.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.798+0000 7f40f67fc700 1 -- 172.21.15.57:0/538408384 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f40f00cd9e0 con 0x7f4100074750 2023-07-20T18:58:46.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.798+0000 7f410539b700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40ec0a1150 0x7f40ec0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:46.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.798+0000 7f410539b700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40ec0a1150 0x7f40ec0a3610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009c60 tx=0x7f40fc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:46.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:46.802+0000 7f40f67fc700 1 -- 172.21.15.57:0/538408384 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f40f0096f70 con 0x7f4100074750 2023-07-20T18:58:47.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.017+0000 7f41075ff700 1 -- 172.21.15.57:0/538408384 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4100061420 con 0x7f4100074750 2023-07-20T18:58:47.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.017+0000 7f40f67fc700 1 -- 172.21.15.57:0/538408384 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7f40f009a590 con 0x7f4100074750 2023-07-20T18:58:47.067 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-07-20T18:58:47.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.019+0000 7f41075ff700 1 -- 172.21.15.57:0/538408384 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40ec0a1150 msgr2=0x7f40ec0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.019+0000 7f41075ff700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40ec0a1150 0x7f40ec0a3610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009c60 tx=0x7f40fc009260 comp rx=0 tx=0).stop 2023-07-20T18:58:47.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.019+0000 7f41075ff700 1 -- 172.21.15.57:0/538408384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4100074750 msgr2=0x7f4100198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.019+0000 7f41075ff700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4100074750 0x7f4100198c90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f40f0005270 tx=0x7f40f00052a0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.019+0000 7f41075ff700 1 -- 172.21.15.57:0/538408384 shutdown_connections 2023-07-20T18:58:47.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.019+0000 7f41075ff700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40ec0a1150 0x7f40ec0a3610 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.019+0000 7f41075ff700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4100073170 0x7f4100198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.020+0000 7f41075ff700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4100108630 0x7f410019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.020+0000 7f41075ff700 1 --2- 172.21.15.57:0/538408384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4100074750 0x7f4100198c90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.020+0000 7f41075ff700 1 -- 172.21.15.57:0/538408384 >> 172.21.15.57:0/538408384 conn(0x7f41000fbcb0 msgr2=0x7f41000fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:47.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.020+0000 7f41075ff700 1 -- 172.21.15.57:0/538408384 shutdown_connections 2023-07-20T18:58:47.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.020+0000 7f41075ff700 1 -- 172.21.15.57:0/538408384 wait complete. 2023-07-20T18:58:47.087 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-20T18:58:47.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:46 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1046594995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:58:47.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:46 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2925841973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T18:58:47.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.245+0000 7f4080bff700 1 -- 172.21.15.57:0/82382196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f407c100760 msgr2=0x7f407c100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.245+0000 7f4080bff700 1 --2- 172.21.15.57:0/82382196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f407c100760 0x7f407c100b80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f406c0099c0 tx=0x7f406c009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.245+0000 7f4080bff700 1 -- 172.21.15.57:0/82382196 shutdown_connections 2023-07-20T18:58:47.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.245+0000 7f4080bff700 1 --2- 172.21.15.57:0/82382196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f407c1022f0 0x7f407c108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.245+0000 7f4080bff700 1 --2- 172.21.15.57:0/82382196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f407c100760 0x7f407c100b80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.245+0000 7f4080bff700 1 --2- 172.21.15.57:0/82382196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f407c101950 0x7f407c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.245+0000 7f4080bff700 1 -- 172.21.15.57:0/82382196 >> 172.21.15.57:0/82382196 conn(0x7f407c0fbcb0 msgr2=0x7f407c0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:47.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.245+0000 7f4080bff700 1 -- 172.21.15.57:0/82382196 shutdown_connections 2023-07-20T18:58:47.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.245+0000 7f4080bff700 1 -- 172.21.15.57:0/82382196 wait complete. 2023-07-20T18:58:47.351 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:46 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1046594995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:58:47.351 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:46 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2925841973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T18:58:47.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.246+0000 7f4080bff700 1 Processor -- start 2023-07-20T18:58:47.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.246+0000 7f4080bff700 1 -- start start 2023-07-20T18:58:47.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.246+0000 7f4080bff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f407c100760 0x7f407c074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:47.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.246+0000 7f4080bff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f407c101950 0x7f407c072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:47.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.246+0000 7f4080bff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f407c1022f0 0x7f407c0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:47.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f4080bff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c073840 con 0x7f407c101950 2023-07-20T18:58:47.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f4080bff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c0739b0 con 0x7f407c100760 2023-07-20T18:58:47.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f4080bff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c1a05c0 con 0x7f407c1022f0 2023-07-20T18:58:47.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f4073fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f407c101950 0x7f407c072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:47.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f407a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f407c100760 0x7f407c074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:47.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f4073fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f407c101950 0x7f407c072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54856/0 (socket says 172.21.15.57:54856) 2023-07-20T18:58:47.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f4073fff700 1 -- 172.21.15.57:0/2567662793 learned_addr learned my addr 172.21.15.57:0/2567662793 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:47.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f407ad9d700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f407c1022f0 0x7f407c0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:47.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f407a59c700 1 -- 172.21.15.57:0/2567662793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f407c1022f0 msgr2=0x7f407c0730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f407a59c700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f407c1022f0 0x7f407c0730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f407a59c700 1 -- 172.21.15.57:0/2567662793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f407c101950 msgr2=0x7f407c072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f407a59c700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f407c101950 0x7f407c072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f407a59c700 1 -- 172.21.15.57:0/2567662793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c0095f0 con 0x7f407c100760 2023-07-20T18:58:47.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.247+0000 7f4073fff700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f407c101950 0x7f407c072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:47.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.248+0000 7f407a59c700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f407c100760 0x7f407c074520 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f406c0098e0 tx=0x7f406c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:47.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.248+0000 7f40737fe700 1 -- 172.21.15.57:0/2567662793 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c01d070 con 0x7f407c100760 2023-07-20T18:58:47.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.248+0000 7f4080bff700 1 -- 172.21.15.57:0/2567662793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407c1a0840 con 0x7f407c100760 2023-07-20T18:58:47.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.248+0000 7f40737fe700 1 -- 172.21.15.57:0/2567662793 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f406c0040a0 con 0x7f407c100760 2023-07-20T18:58:47.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.248+0000 7f40737fe700 1 -- 172.21.15.57:0/2567662793 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c005280 con 0x7f407c100760 2023-07-20T18:58:47.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.248+0000 7f4080bff700 1 -- 172.21.15.57:0/2567662793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407c1a0c60 con 0x7f407c100760 2023-07-20T18:58:47.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.249+0000 7f4080bff700 1 -- 172.21.15.57:0/2567662793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f407c10c100 con 0x7f407c100760 2023-07-20T18:58:47.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.251+0000 7f40737fe700 1 -- 172.21.15.57:0/2567662793 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f406c004870 con 0x7f407c100760 2023-07-20T18:58:47.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.253+0000 7f40737fe700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40680a1150 0x7f40680a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:47.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.253+0000 7f40737fe700 1 -- 172.21.15.57:0/2567662793 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f406c0cdd00 con 0x7f407c100760 2023-07-20T18:58:47.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.253+0000 7f4073fff700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40680a1150 0x7f40680a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:47.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.253+0000 7f4073fff700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40680a1150 0x7f40680a3610 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4064005950 tx=0x7f406400b500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:47.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.255+0000 7f40737fe700 1 -- 172.21.15.57:0/2567662793 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f406c097290 con 0x7f407c100760 2023-07-20T18:58:47.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:46 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1046594995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:58:47.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:46 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2925841973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T18:58:47.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.465+0000 7f4080bff700 1 -- 172.21.15.57:0/2567662793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f407c1a1330 con 0x7f407c100760 2023-07-20T18:58:47.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.466+0000 7f40737fe700 1 -- 172.21.15.57:0/2567662793 <== mon.2 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v86) v1 ==== 103+0+11 (secure 0 0 0) 0x7f406c09a8b0 con 0x7f407c100760 2023-07-20T18:58:47.516 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-07-20T18:58:47.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.468+0000 7f4080bff700 1 -- 172.21.15.57:0/2567662793 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40680a1150 msgr2=0x7f40680a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.468+0000 7f4080bff700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40680a1150 0x7f40680a3610 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4064005950 tx=0x7f406400b500 comp rx=0 tx=0).stop 2023-07-20T18:58:47.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.468+0000 7f4080bff700 1 -- 172.21.15.57:0/2567662793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f407c100760 msgr2=0x7f407c074520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.468+0000 7f4080bff700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f407c100760 0x7f407c074520 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f406c0098e0 tx=0x7f406c004730 comp rx=0 tx=0).stop 2023-07-20T18:58:47.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.468+0000 7f4080bff700 1 -- 172.21.15.57:0/2567662793 shutdown_connections 2023-07-20T18:58:47.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.468+0000 7f4080bff700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f40680a1150 0x7f40680a3610 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.468+0000 7f4080bff700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f407c1022f0 0x7f407c0730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.468+0000 7f4080bff700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f407c100760 0x7f407c074520 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.468+0000 7f4080bff700 1 --2- 172.21.15.57:0/2567662793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f407c101950 0x7f407c072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.468+0000 7f4080bff700 1 -- 172.21.15.57:0/2567662793 >> 172.21.15.57:0/2567662793 conn(0x7f407c0fbcb0 msgr2=0x7f407c0fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:47.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.469+0000 7f4080bff700 1 -- 172.21.15.57:0/2567662793 shutdown_connections 2023-07-20T18:58:47.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.469+0000 7f4080bff700 1 -- 172.21.15.57:0/2567662793 wait complete. 2023-07-20T18:58:47.536 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-20T18:58:47.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.692+0000 7f85193ff700 1 -- 172.21.15.57:0/3888005069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8514101950 msgr2=0x7f8514101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.692+0000 7f85193ff700 1 --2- 172.21.15.57:0/3888005069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8514101950 0x7f8514101db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f85080098f0 tx=0x7f8508009c00 comp rx=0 tx=0).stop 2023-07-20T18:58:47.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.693+0000 7f85193ff700 1 -- 172.21.15.57:0/3888005069 shutdown_connections 2023-07-20T18:58:47.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.693+0000 7f85193ff700 1 --2- 172.21.15.57:0/3888005069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85141022f0 0x7f8514108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.693+0000 7f85193ff700 1 --2- 172.21.15.57:0/3888005069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8514100760 0x7f8514100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.693+0000 7f85193ff700 1 --2- 172.21.15.57:0/3888005069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8514101950 0x7f8514101db0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.693+0000 7f85193ff700 1 -- 172.21.15.57:0/3888005069 >> 172.21.15.57:0/3888005069 conn(0x7f85140fbcb0 msgr2=0x7f85140fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:47.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.693+0000 7f85193ff700 1 -- 172.21.15.57:0/3888005069 shutdown_connections 2023-07-20T18:58:47.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.693+0000 7f85193ff700 1 -- 172.21.15.57:0/3888005069 wait complete. 2023-07-20T18:58:47.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.694+0000 7f85193ff700 1 Processor -- start 2023-07-20T18:58:47.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.694+0000 7f85193ff700 1 -- start start 2023-07-20T18:58:47.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85193ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8514100760 0x7f8514196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:47.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85193ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8514101950 0x7f8514196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:47.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85193ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85141022f0 0x7f851419dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:47.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85193ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851419e190 con 0x7f85141022f0 2023-07-20T18:58:47.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85193ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851419e2d0 con 0x7f8514100760 2023-07-20T18:58:47.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85193ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851419e460 con 0x7f8514101950 2023-07-20T18:58:47.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85137fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85141022f0 0x7f851419dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:47.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85137fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85141022f0 0x7f851419dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54872/0 (socket says 172.21.15.57:54872) 2023-07-20T18:58:47.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f8512ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8514100760 0x7f8514196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:47.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85137fe700 1 -- 172.21.15.57:0/3416327436 learned_addr learned my addr 172.21.15.57:0/3416327436 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:47.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85127fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8514101950 0x7f8514196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:47.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85137fe700 1 -- 172.21.15.57:0/3416327436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8514101950 msgr2=0x7f8514196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85137fe700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8514101950 0x7f8514196b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85137fe700 1 -- 172.21.15.57:0/3416327436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8514100760 msgr2=0x7f8514196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85137fe700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8514100760 0x7f8514196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.695+0000 7f85137fe700 1 -- 172.21.15.57:0/3416327436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c009d00 con 0x7f85141022f0 2023-07-20T18:58:47.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.696+0000 7f85137fe700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85141022f0 0x7f851419dbc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f850c00e3c0 tx=0x7f850c00e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:47.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.696+0000 7f84fbfff700 1 -- 172.21.15.57:0/3416327436 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c0090d0 con 0x7f85141022f0 2023-07-20T18:58:47.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.696+0000 7f84fbfff700 1 -- 172.21.15.57:0/3416327436 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f850c00f040 con 0x7f85141022f0 2023-07-20T18:58:47.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.696+0000 7f85193ff700 1 -- 172.21.15.57:0/3416327436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8508009520 con 0x7f85141022f0 2023-07-20T18:58:47.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.696+0000 7f84fbfff700 1 -- 172.21.15.57:0/3416327436 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c007c30 con 0x7f85141022f0 2023-07-20T18:58:47.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.696+0000 7f85193ff700 1 -- 172.21.15.57:0/3416327436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851419eac0 con 0x7f85141022f0 2023-07-20T18:58:47.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.697+0000 7f85193ff700 1 -- 172.21.15.57:0/3416327436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8514107d40 con 0x7f85141022f0 2023-07-20T18:58:47.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.699+0000 7f84fbfff700 1 -- 172.21.15.57:0/3416327436 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f850c003a00 con 0x7f85141022f0 2023-07-20T18:58:47.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.701+0000 7f84fbfff700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f84fc0a11d0 0x7f84fc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:47.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.701+0000 7f84fbfff700 1 -- 172.21.15.57:0/3416327436 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f850c0cc900 con 0x7f85141022f0 2023-07-20T18:58:47.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.701+0000 7f8512ffd700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f84fc0a11d0 0x7f84fc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:47.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.702+0000 7f8512ffd700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f84fc0a11d0 0x7f84fc0a3690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85141017b0 tx=0x7f8504009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:47.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.704+0000 7f84fbfff700 1 -- 172.21.15.57:0/3416327436 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f850c095ef0 con 0x7f85141022f0 2023-07-20T18:58:47.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.931+0000 7f85193ff700 1 -- 172.21.15.57:0/3416327436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f851402ce00 con 0x7f85141022f0 2023-07-20T18:58:47.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.932+0000 7f84fbfff700 1 -- 172.21.15.57:0/3416327436 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) v1 ==== 81+0+589 (secure 0 0 0) 0x7f850c099510 con 0x7f85141022f0 2023-07-20T18:58:47.982 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/2188156324 2023-07-21T18:52:02.610729+0000 2023-07-20T18:58:47.982 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3804015220 2023-07-21T18:52:02.610729+0000 2023-07-20T18:58:47.982 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/195678005 2023-07-21T18:51:34.614904+0000 2023-07-20T18:58:47.982 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/1582727629 2023-07-21T18:51:34.614904+0000 2023-07-20T18:58:47.982 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/2188156324 2023-07-21T18:52:02.610729+0000 2023-07-20T18:58:47.982 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/2044626900 2023-07-21T18:51:34.614904+0000 2023-07-20T18:58:47.982 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/748511752 2023-07-21T18:51:34.614904+0000 2023-07-20T18:58:47.983 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/1151100650 2023-07-21T18:52:02.610729+0000 2023-07-20T18:58:47.983 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/195678005 2023-07-21T18:51:34.614904+0000 2023-07-20T18:58:47.983 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3446341201 2023-07-21T18:52:02.610729+0000 2023-07-20T18:58:47.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 -- 172.21.15.57:0/3416327436 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f84fc0a11d0 msgr2=0x7f84fc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f84fc0a11d0 0x7f84fc0a3690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85141017b0 tx=0x7f8504009260 comp rx=0 tx=0).stop 2023-07-20T18:58:47.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 -- 172.21.15.57:0/3416327436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85141022f0 msgr2=0x7f851419dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:47.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85141022f0 0x7f851419dbc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f850c00e3c0 tx=0x7f850c00e6d0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 -- 172.21.15.57:0/3416327436 shutdown_connections 2023-07-20T18:58:47.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f84fc0a11d0 0x7f84fc0a3690 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85141022f0 0x7f851419dbc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8514100760 0x7f8514196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 --2- 172.21.15.57:0/3416327436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8514101950 0x7f8514196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:47.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 -- 172.21.15.57:0/3416327436 >> 172.21.15.57:0/3416327436 conn(0x7f85140fbcb0 msgr2=0x7f8514104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:47.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 -- 172.21.15.57:0/3416327436 shutdown_connections 2023-07-20T18:58:47.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:47.934+0000 7f85193ff700 1 -- 172.21.15.57:0/3416327436 wait complete. 2023-07-20T18:58:47.985 INFO:teuthology.orchestra.run.smithi057.stderr:listed 10 entries 2023-07-20T18:58:48.001 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-07-20T18:58:48.002 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-20T18:58:48.046 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T18:58:48.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:47 smithi057 ceph-mon[126340]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:48.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:47 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/538408384' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T18:58:48.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:47 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2567662793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T18:58:48.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.241+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3048114337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75a8102300 msgr2=0x7f75a8108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:48.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.241+0000 7f75ad5ff700 1 --2- 172.21.15.57:0/3048114337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75a8102300 0x7f75a8108bb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f75a000b0d0 tx=0x7f75a000b3e0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.241+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3048114337 shutdown_connections 2023-07-20T18:58:48.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.241+0000 7f75ad5ff700 1 --2- 172.21.15.57:0/3048114337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75a8100770 0x7f75a8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.241+0000 7f75ad5ff700 1 --2- 172.21.15.57:0/3048114337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75a8102300 0x7f75a8108bb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.241+0000 7f75ad5ff700 1 --2- 172.21.15.57:0/3048114337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a8101960 0x7f75a8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.241+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3048114337 >> 172.21.15.57:0/3048114337 conn(0x7f75a80fbca0 msgr2=0x7f75a80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:48.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.241+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3048114337 shutdown_connections 2023-07-20T18:58:48.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.241+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3048114337 wait complete. 2023-07-20T18:58:48.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.242+0000 7f75ad5ff700 1 Processor -- start 2023-07-20T18:58:48.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.242+0000 7f75ad5ff700 1 -- start start 2023-07-20T18:58:48.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75ad5ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75a8100770 0x7f75a81965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:48.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75ad5ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75a8101960 0x7f75a8196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:48.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75ad5ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a8102300 0x7f75a819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:48.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75ad5ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a819e130 con 0x7f75a8100770 2023-07-20T18:58:48.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75ad5ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a819e270 con 0x7f75a8102300 2023-07-20T18:58:48.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75ad5ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a819e400 con 0x7f75a8101960 2023-07-20T18:58:48.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75a6ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75a8100770 0x7f75a81965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:48.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75a6ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75a8100770 0x7f75a81965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54896/0 (socket says 172.21.15.57:54896) 2023-07-20T18:58:48.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75a6ffd700 1 -- 172.21.15.57:0/3069546821 learned_addr learned my addr 172.21.15.57:0/3069546821 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:48.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75a77fe700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a8102300 0x7f75a819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:48.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75a67fc700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75a8101960 0x7f75a8196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:48.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75a6ffd700 1 -- 172.21.15.57:0/3069546821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75a8101960 msgr2=0x7f75a8196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:48.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75a6ffd700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75a8101960 0x7f75a8196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75a6ffd700 1 -- 172.21.15.57:0/3069546821 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a8102300 msgr2=0x7f75a819db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:48.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75a6ffd700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a8102300 0x7f75a819db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.243+0000 7f75a6ffd700 1 -- 172.21.15.57:0/3069546821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0009d00 con 0x7f75a8100770 2023-07-20T18:58:48.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.244+0000 7f75a6ffd700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75a8100770 0x7f75a81965b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f759800d980 tx=0x7f759800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:48.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.244+0000 7f758ffff700 1 -- 172.21.15.57:0/3069546821 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598004020 con 0x7f75a8100770 2023-07-20T18:58:48.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.244+0000 7f758ffff700 1 -- 172.21.15.57:0/3069546821 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f759800f690 con 0x7f75a8100770 2023-07-20T18:58:48.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.244+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3069546821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a819e6e0 con 0x7f75a8100770 2023-07-20T18:58:48.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.244+0000 7f758ffff700 1 -- 172.21.15.57:0/3069546821 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598003b40 con 0x7f75a8100770 2023-07-20T18:58:48.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.244+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3069546821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a819ec40 con 0x7f75a8100770 2023-07-20T18:58:48.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.245+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3069546821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a810c0d0 con 0x7f75a8100770 2023-07-20T18:58:48.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.247+0000 7f758ffff700 1 -- 172.21.15.57:0/3069546821 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f759800fb20 con 0x7f75a8100770 2023-07-20T18:58:48.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.249+0000 7f758ffff700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f75900a1220 0x7f75900a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:48.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.249+0000 7f758ffff700 1 -- 172.21.15.57:0/3069546821 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f75980cc6e0 con 0x7f75a8100770 2023-07-20T18:58:48.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.249+0000 7f75a67fc700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f75900a1220 0x7f75900a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:48.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.250+0000 7f75a67fc700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f75900a1220 0x7f75900a36e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f759c009c60 tx=0x7f759c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:48.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.253+0000 7f758ffff700 1 -- 172.21.15.57:0/3069546821 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7598095be0 con 0x7f75a8100770 2023-07-20T18:58:48.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:47 smithi192 ceph-mon[129200]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:48.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:47 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/538408384' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T18:58:48.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:47 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2567662793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T18:58:48.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:47 smithi088 ceph-mon[128823]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:48.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:47 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/538408384' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T18:58:48.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:47 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2567662793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T18:58:48.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.464+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3069546821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75a8107530 con 0x7f75a8100770 2023-07-20T18:58:48.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.465+0000 7f758ffff700 1 -- 172.21.15.57:0/3069546821 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23911 (secure 0 0 0) 0x7f7598099200 con 0x7f75a8100770 2023-07-20T18:58:48.515 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:48.516 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":86,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","created":"2023-07-20T18:51:01.245641+0000","modified":"2023-07-20T18:58:35.383850+0000","last_up_change":"2023-07-20T18:57:44.823544+0000","last_in_change":"2023-07-20T18:57:27.428782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T18:54:23.215777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T18:58:28.447992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T18:58:29.772740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T18:58:33.395093+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"627a67ea-1366-4b3f-babd-067f75bad1e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6803","nonce":3023441607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6805","nonce":3023441607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6809","nonce":3023441607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6807","nonce":3023441607}]},"public_addr":"172.21.15.57:6803/3023441607","cluster_addr":"172.21.15.57:6805/3023441607","heartbeat_back_addr":"172.21.15.57:6809/3023441607","heartbeat_front_addr":"172.21.15.57:6807/3023441607","state":["exists","up"]},{"osd":1,"uuid":"adcdb236-334e-4478-98a8-7d0628492082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6801","nonce":1809528555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6803","nonce":1809528555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6807","nonce":1809528555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6805","nonce":1809528555}]},"public_addr":"172.21.15.88:6801/1809528555","cluster_addr":"172.21.15.88:6803/1809528555","heartbeat_back_addr":"172.21.15.88:6807/1809528555","heartbeat_front_addr":"172.21.15.88:6805/1809528555","state":["exists","up"]},{"osd":2,"uuid":"3cd56229-2067-433a-b9f9-f47b1b878b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6801","nonce":3891687382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6803","nonce":3891687382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6807","nonce":3891687382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6805","nonce":3891687382}]},"public_addr":"172.21.15.192:6801/3891687382","cluster_addr":"172.21.15.192:6803/3891687382","heartbeat_back_addr":"172.21.15.192:6807/3891687382","heartbeat_front_addr":"172.21.15.192:6805/3891687382","state":["exists","up"]},{"osd":3,"uuid":"aae250ab-e242-4ef8-9c3a-a4fa36102c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6811","nonce":2920717314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6813","nonce":2920717314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6817","nonce":2920717314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6815","nonce":2920717314}]},"public_addr":"172.21.15.57:6811/2920717314","cluster_addr":"172.21.15.57:6813/2920717314","heartbeat_back_addr":"172.21.15.57:6817/2920717314","heartbeat_front_addr":"172.21.15.57:6815/2920717314","state":["exists","up"]},{"osd":4,"uuid":"8c8d1dfa-1cff-4f0a-ae72-22bd522ed696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6809","nonce":3911525610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6811","nonce":3911525610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6815","nonce":3911525610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6813","nonce":3911525610}]},"public_addr":"172.21.15.88:6809/3911525610","cluster_addr":"172.21.15.88:6811/3911525610","heartbeat_back_addr":"172.21.15.88:6815/3911525610","heartbeat_front_addr":"172.21.15.88:6813/3911525610","state":["exists","up"]},{"osd":5,"uuid":"5a2a56ab-2af9-4176-ba4d-b2138f0e7c75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6809","nonce":905436397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6811","nonce":905436397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6815","nonce":905436397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6813","nonce":905436397}]},"public_addr":"172.21.15.192:6809/905436397","cluster_addr":"172.21.15.192:6811/905436397","heartbeat_back_addr":"172.21.15.192:6815/905436397","heartbeat_front_addr":"172.21.15.192:6813/905436397","state":["exists","up"]},{"osd":6,"uuid":"03eafdda-278c-443c-b376-677eeb50262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6819","nonce":333894423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6821","nonce":333894423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6825","nonce":333894423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6823","nonce":333894423}]},"public_addr":"172.21.15.57:6819/333894423","cluster_addr":"172.21.15.57:6821/333894423","heartbeat_back_addr":"172.21.15.57:6825/333894423","heartbeat_front_addr":"172.21.15.57:6823/333894423","state":["exists","up"]},{"osd":7,"uuid":"3f01254e-be73-41c8-b979-3b6a7534e2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6817","nonce":3031485551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6819","nonce":3031485551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6823","nonce":3031485551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6821","nonce":3031485551}]},"public_addr":"172.21.15.88:6817/3031485551","cluster_addr":"172.21.15.88:6819/3031485551","heartbeat_back_addr":"172.21.15.88:6823/3031485551","heartbeat_front_addr":"172.21.15.88:6821/3031485551","state":["exists","up"]},{"osd":8,"uuid":"624a150e-f47b-4a76-8785-c818dd65342e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6817","nonce":918056390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6819","nonce":918056390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6823","nonce":918056390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6821","nonce":918056390}]},"public_addr":"172.21.15.192:6817/918056390","cluster_addr":"172.21.15.192:6819/918056390","heartbeat_back_addr":"172.21.15.192:6823/918056390","heartbeat_front_addr":"172.21.15.192:6821/918056390","state":["exists","up"]},{"osd":9,"uuid":"6d9cb69d-ed7b-4009-9ec5-027becc577d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6827","nonce":1773340763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6829","nonce":1773340763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6833","nonce":1773340763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6831","nonce":1773340763}]},"public_addr":"172.21.15.57:6827/1773340763","cluster_addr":"172.21.15.57:6829/1773340763","heartbeat_back_addr":"172.21.15.57:6833/1773340763","heartbeat_front_addr":"172.21.15.57:6831/1773340763","state":["exists","up"]},{"osd":10,"uuid":"d5799219-0979-40b1-a596-acd37f37a72d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6825","nonce":2781975579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6827","nonce":2781975579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6831","nonce":2781975579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6829","nonce":2781975579}]},"public_addr":"172.21.15.88:6825/2781975579","cluster_addr":"172.21.15.88:6827/2781975579","heartbeat_back_addr":"172.21.15.88:6831/2781975579","heartbeat_front_addr":"172.21.15.88:6829/2781975579","state":["exists","up"]},{"osd":11,"uuid":"e77cac6c-107e-41f9-8efd-e1140c4e3bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6825","nonce":2864190031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6827","nonce":2864190031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6831","nonce":2864190031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6829","nonce":2864190031}]},"public_addr":"172.21.15.192:6825/2864190031","cluster_addr":"172.21.15.192:6827/2864190031","heartbeat_back_addr":"172.21.15.192:6831/2864190031","heartbeat_front_addr":"172.21.15.192:6829/2864190031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:53:38.180971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:00.253039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:20.776122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:48.631094+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:09.288231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:29.595802+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:55.089126+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:15.637849+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:35.985101+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:01.544767+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:22.351743+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:43.255157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6801/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/3804015220":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6800/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1582727629":"2023-07-21T18:51:34.614904+0000","172.21.15.57:6800/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/2044626900":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/748511752":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1151100650":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6801/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/3446341201":"2023-07-21T18:52:02.610729+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T18:58:48.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3069546821 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f75900a1220 msgr2=0x7f75900a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:48.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f75900a1220 0x7f75900a36e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f759c009c60 tx=0x7f759c009260 comp rx=0 tx=0).stop 2023-07-20T18:58:48.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3069546821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75a8100770 msgr2=0x7f75a81965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:48.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75a8100770 0x7f75a81965b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f759800d980 tx=0x7f759800dc90 comp rx=0 tx=0).stop 2023-07-20T18:58:48.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3069546821 shutdown_connections 2023-07-20T18:58:48.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f75900a1220 0x7f75900a36e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a8102300 0x7f75a819db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75a8100770 0x7f75a81965b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 --2- 172.21.15.57:0/3069546821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75a8101960 0x7f75a8196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3069546821 >> 172.21.15.57:0/3069546821 conn(0x7f75a80fbca0 msgr2=0x7f75a80fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:48.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3069546821 shutdown_connections 2023-07-20T18:58:48.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.468+0000 7f75ad5ff700 1 -- 172.21.15.57:0/3069546821 wait complete. 2023-07-20T18:58:48.536 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T18:54:23.215777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-20T18:58:28.447992+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-20T18:58:29.772740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-20T18:58:33.395093+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-20T18:58:48.537 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T18:58:48.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.693+0000 7fe3d39ff700 1 -- 172.21.15.57:0/2794243532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc103b10 msgr2=0x7fe3cc105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:48.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.693+0000 7fe3d39ff700 1 --2- 172.21.15.57:0/2794243532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc103b10 0x7fe3cc105f00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009990 tx=0x7fe3bc009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.693+0000 7fe3d39ff700 1 -- 172.21.15.57:0/2794243532 shutdown_connections 2023-07-20T18:58:48.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.693+0000 7fe3d39ff700 1 --2- 172.21.15.57:0/2794243532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3cc106440 0x7fe3cc1088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.693+0000 7fe3d39ff700 1 --2- 172.21.15.57:0/2794243532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1011e0 0x7fe3cc1035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.693+0000 7fe3d39ff700 1 --2- 172.21.15.57:0/2794243532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc103b10 0x7fe3cc105f00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.693+0000 7fe3d39ff700 1 -- 172.21.15.57:0/2794243532 >> 172.21.15.57:0/2794243532 conn(0x7fe3cc0fabd0 msgr2=0x7fe3cc0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:48.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.693+0000 7fe3d39ff700 1 -- 172.21.15.57:0/2794243532 shutdown_connections 2023-07-20T18:58:48.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.693+0000 7fe3d39ff700 1 -- 172.21.15.57:0/2794243532 wait complete. 2023-07-20T18:58:48.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.694+0000 7fe3d39ff700 1 Processor -- start 2023-07-20T18:58:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.694+0000 7fe3d39ff700 1 -- start start 2023-07-20T18:58:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d39ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1011e0 0x7fe3cc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d39ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3cc103b10 0x7fe3cc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d39ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106440 0x7fe3cc19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d39ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc1a0250 con 0x7fe3cc106440 2023-07-20T18:58:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d39ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc1a0390 con 0x7fe3cc1011e0 2023-07-20T18:58:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d39ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc1a0520 con 0x7fe3cc103b10 2023-07-20T18:58:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d1f9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106440 0x7fe3cc19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d1f9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106440 0x7fe3cc19fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54912/0 (socket says 172.21.15.57:54912) 2023-07-20T18:58:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d1f9c700 1 -- 172.21.15.57:0/1668674242 learned_addr learned my addr 172.21.15.57:0/1668674242 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d179b700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1011e0 0x7fe3cc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d0f9a700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3cc103b10 0x7fe3cc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d1f9c700 1 -- 172.21.15.57:0/1668674242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3cc103b10 msgr2=0x7fe3cc198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d1f9c700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3cc103b10 0x7fe3cc198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d1f9c700 1 -- 172.21.15.57:0/1668674242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1011e0 msgr2=0x7fe3cc1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d1f9c700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1011e0 0x7fe3cc1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d1f9c700 1 -- 172.21.15.57:0/1668674242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0095f0 con 0x7fe3cc106440 2023-07-20T18:58:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.695+0000 7fe3d179b700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1011e0 0x7fe3cc1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.696+0000 7fe3d1f9c700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106440 0x7fe3cc19fc80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400ef30 tx=0x7fe3c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.696+0000 7fe3c27fc700 1 -- 172.21.15.57:0/1668674242 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4018070 con 0x7fe3cc106440 2023-07-20T18:58:48.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.696+0000 7fe3d39ff700 1 -- 172.21.15.57:0/1668674242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc1a07a0 con 0x7fe3cc106440 2023-07-20T18:58:48.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.696+0000 7fe3c27fc700 1 -- 172.21.15.57:0/1668674242 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fe3c400f040 con 0x7fe3cc106440 2023-07-20T18:58:48.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.696+0000 7fe3c27fc700 1 -- 172.21.15.57:0/1668674242 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400cd50 con 0x7fe3cc106440 2023-07-20T18:58:48.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.696+0000 7fe3d39ff700 1 -- 172.21.15.57:0/1668674242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc1a0c60 con 0x7fe3cc106440 2023-07-20T18:58:48.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.697+0000 7fe3d39ff700 1 -- 172.21.15.57:0/1668674242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc10c0e0 con 0x7fe3cc106440 2023-07-20T18:58:48.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.699+0000 7fe3c27fc700 1 -- 172.21.15.57:0/1668674242 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe3c4003a00 con 0x7fe3cc106440 2023-07-20T18:58:48.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.700+0000 7fe3c27fc700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe3b80a1150 0x7fe3b80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:48.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.701+0000 7fe3c27fc700 1 -- 172.21.15.57:0/1668674242 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe3c40cd0b0 con 0x7fe3cc106440 2023-07-20T18:58:48.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.701+0000 7fe3d179b700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe3b80a1150 0x7fe3b80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:48.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.701+0000 7fe3d179b700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe3b80a1150 0x7fe3b80a3610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8011490 tx=0x7fe3c8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:48.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.703+0000 7fe3c27fc700 1 -- 172.21.15.57:0/1668674242 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe3c4096640 con 0x7fe3cc106440 2023-07-20T18:58:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.914+0000 7fe3d39ff700 1 -- 172.21.15.57:0/1668674242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe3cc1a12a0 con 0x7fe3cc106440 2023-07-20T18:58:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.914+0000 7fe3c27fc700 1 -- 172.21.15.57:0/1668674242 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe3c4099c60 con 0x7fe3cc106440 2023-07-20T18:58:48.964 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-07-20T18:58:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 -- 172.21.15.57:0/1668674242 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe3b80a1150 msgr2=0x7fe3b80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe3b80a1150 0x7fe3b80a3610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8011490 tx=0x7fe3c8009260 comp rx=0 tx=0).stop 2023-07-20T18:58:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 -- 172.21.15.57:0/1668674242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106440 msgr2=0x7fe3cc19fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106440 0x7fe3cc19fc80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400ef30 tx=0x7fe3c400c3a0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 -- 172.21.15.57:0/1668674242 shutdown_connections 2023-07-20T18:58:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fe3b80a1150 0x7fe3b80a3610 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106440 0x7fe3cc19fc80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1011e0 0x7fe3cc1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 --2- 172.21.15.57:0/1668674242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3cc103b10 0x7fe3cc198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 -- 172.21.15.57:0/1668674242 >> 172.21.15.57:0/1668674242 conn(0x7fe3cc0fabd0 msgr2=0x7fe3cc102630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 -- 172.21.15.57:0/1668674242 shutdown_connections 2023-07-20T18:58:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:48.917+0000 7fe3d39ff700 1 -- 172.21.15.57:0/1668674242 wait complete. 2023-07-20T18:58:48.985 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-20T18:58:49.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.142+0000 7fca697ff700 1 -- 172.21.15.57:0/2193937127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca641022f0 msgr2=0x7fca64108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.142+0000 7fca697ff700 1 --2- 172.21.15.57:0/2193937127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca641022f0 0x7fca64108ba0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00b0d0 tx=0x7fca5c00b3e0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.143+0000 7fca697ff700 1 -- 172.21.15.57:0/2193937127 shutdown_connections 2023-07-20T18:58:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.143+0000 7fca697ff700 1 --2- 172.21.15.57:0/2193937127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca641022f0 0x7fca64108ba0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.143+0000 7fca697ff700 1 --2- 172.21.15.57:0/2193937127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca64100760 0x7fca64100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.143+0000 7fca697ff700 1 --2- 172.21.15.57:0/2193937127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca64101950 0x7fca64101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.143+0000 7fca697ff700 1 -- 172.21.15.57:0/2193937127 >> 172.21.15.57:0/2193937127 conn(0x7fca640fbcb0 msgr2=0x7fca640fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.143+0000 7fca697ff700 1 -- 172.21.15.57:0/2193937127 shutdown_connections 2023-07-20T18:58:49.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.143+0000 7fca697ff700 1 -- 172.21.15.57:0/2193937127 wait complete. 2023-07-20T18:58:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca697ff700 1 Processor -- start 2023-07-20T18:58:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca697ff700 1 -- start start 2023-07-20T18:58:49.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca697ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca64100760 0x7fca64198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:49.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca697ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca64101950 0x7fca64198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:49.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca697ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca641022f0 0x7fca6419fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:49.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca697ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca641a03d0 con 0x7fca64101950 2023-07-20T18:58:49.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca697ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca641a0510 con 0x7fca64100760 2023-07-20T18:58:49.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca697ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca641a06a0 con 0x7fca641022f0 2023-07-20T18:58:49.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca627fc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca64101950 0x7fca64198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca627fc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca64101950 0x7fca64198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54928/0 (socket says 172.21.15.57:54928) 2023-07-20T18:58:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.144+0000 7fca627fc700 1 -- 172.21.15.57:0/882233511 learned_addr learned my addr 172.21.15.57:0/882233511 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.145+0000 7fca62ffd700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca64100760 0x7fca64198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.145+0000 7fca637fe700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca641022f0 0x7fca6419fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.145+0000 7fca627fc700 1 -- 172.21.15.57:0/882233511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca641022f0 msgr2=0x7fca6419fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.145+0000 7fca627fc700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca641022f0 0x7fca6419fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.145+0000 7fca627fc700 1 -- 172.21.15.57:0/882233511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca64100760 msgr2=0x7fca64198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:49.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.145+0000 7fca627fc700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca64100760 0x7fca64198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.145+0000 7fca627fc700 1 -- 172.21.15.57:0/882233511 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c009d00 con 0x7fca64101950 2023-07-20T18:58:49.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.145+0000 7fca62ffd700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca64100760 0x7fca64198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:49.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.145+0000 7fca627fc700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca64101950 0x7fca64198d90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fca5800d950 tx=0x7fca5800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:49.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.145+0000 7fca4bfff700 1 -- 172.21.15.57:0/882233511 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5800f890 con 0x7fca64101950 2023-07-20T18:58:49.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.146+0000 7fca4bfff700 1 -- 172.21.15.57:0/882233511 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fca58010460 con 0x7fca64101950 2023-07-20T18:58:49.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.146+0000 7fca697ff700 1 -- 172.21.15.57:0/882233511 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca641a0980 con 0x7fca64101950 2023-07-20T18:58:49.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.146+0000 7fca4bfff700 1 -- 172.21.15.57:0/882233511 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5800e610 con 0x7fca64101950 2023-07-20T18:58:49.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.146+0000 7fca697ff700 1 -- 172.21.15.57:0/882233511 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca641a0ee0 con 0x7fca64101950 2023-07-20T18:58:49.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.147+0000 7fca697ff700 1 -- 172.21.15.57:0/882233511 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6410c110 con 0x7fca64101950 2023-07-20T18:58:49.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.149+0000 7fca4bfff700 1 -- 172.21.15.57:0/882233511 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fca58010960 con 0x7fca64101950 2023-07-20T18:58:49.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.150+0000 7fca4bfff700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fca4c0a11d0 0x7fca4c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:49.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.150+0000 7fca4bfff700 1 -- 172.21.15.57:0/882233511 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fca580cc810 con 0x7fca64101950 2023-07-20T18:58:49.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.150+0000 7fca62ffd700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fca4c0a11d0 0x7fca4c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:49.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.151+0000 7fca62ffd700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fca4c0a11d0 0x7fca4c0a3690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fca641017b0 tx=0x7fca54009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:49.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.154+0000 7fca4bfff700 1 -- 172.21.15.57:0/882233511 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fca58095d70 con 0x7fca64101950 2023-07-20T18:58:49.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:48 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3416327436' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T18:58:49.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:48 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3069546821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:49.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:48 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3416327436' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T18:58:49.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:48 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3069546821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:49.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:48 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3416327436' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T18:58:49.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:48 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3069546821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:49.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.363+0000 7fca697ff700 1 -- 172.21.15.57:0/882233511 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fca641a14f0 con 0x7fca64101950 2023-07-20T18:58:49.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.363+0000 7fca4bfff700 1 -- 172.21.15.57:0/882233511 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7fca58016020 con 0x7fca64101950 2023-07-20T18:58:49.413 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-07-20T18:58:49.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 -- 172.21.15.57:0/882233511 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fca4c0a11d0 msgr2=0x7fca4c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:49.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fca4c0a11d0 0x7fca4c0a3690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fca641017b0 tx=0x7fca54009260 comp rx=0 tx=0).stop 2023-07-20T18:58:49.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 -- 172.21.15.57:0/882233511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca64101950 msgr2=0x7fca64198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:49.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca64101950 0x7fca64198d90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fca5800d950 tx=0x7fca5800dc60 comp rx=0 tx=0).stop 2023-07-20T18:58:49.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 -- 172.21.15.57:0/882233511 shutdown_connections 2023-07-20T18:58:49.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fca4c0a11d0 0x7fca4c0a3690 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca641022f0 0x7fca6419fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca64100760 0x7fca64198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 --2- 172.21.15.57:0/882233511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca64101950 0x7fca64198d90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 -- 172.21.15.57:0/882233511 >> 172.21.15.57:0/882233511 conn(0x7fca640fbcb0 msgr2=0x7fca64107140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:49.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 -- 172.21.15.57:0/882233511 shutdown_connections 2023-07-20T18:58:49.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.366+0000 7fca697ff700 1 -- 172.21.15.57:0/882233511 wait complete. 2023-07-20T18:58:49.433 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-20T18:58:49.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.590+0000 7f7d599ff700 1 -- 172.21.15.57:0/1290229134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d54074750 msgr2=0x7f7d54072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:49.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.590+0000 7f7d599ff700 1 --2- 172.21.15.57:0/1290229134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d54074750 0x7f7d54072ba0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009990 tx=0x7f7d44009ca0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.591+0000 7f7d599ff700 1 -- 172.21.15.57:0/1290229134 shutdown_connections 2023-07-20T18:58:49.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.591+0000 7f7d599ff700 1 --2- 172.21.15.57:0/1290229134 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54108630 0x7f7d5410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.591+0000 7f7d599ff700 1 --2- 172.21.15.57:0/1290229134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d54074750 0x7f7d54072ba0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.591+0000 7f7d599ff700 1 --2- 172.21.15.57:0/1290229134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d54073170 0x7f7d540735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.591+0000 7f7d599ff700 1 -- 172.21.15.57:0/1290229134 >> 172.21.15.57:0/1290229134 conn(0x7f7d540fbcb0 msgr2=0x7f7d540fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:49.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.591+0000 7f7d599ff700 1 -- 172.21.15.57:0/1290229134 shutdown_connections 2023-07-20T18:58:49.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.591+0000 7f7d599ff700 1 -- 172.21.15.57:0/1290229134 wait complete. 2023-07-20T18:58:49.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.592+0000 7f7d599ff700 1 Processor -- start 2023-07-20T18:58:49.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.592+0000 7f7d599ff700 1 -- start start 2023-07-20T18:58:49.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.592+0000 7f7d599ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d54073170 0x7f7d54198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:49.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.592+0000 7f7d599ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d54074750 0x7f7d54198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:49.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.592+0000 7f7d599ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54108630 0x7f7d5419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:49.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.592+0000 7f7d599ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d541a02c0 con 0x7f7d54073170 2023-07-20T18:58:49.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.592+0000 7f7d599ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d541a0400 con 0x7f7d54108630 2023-07-20T18:58:49.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.592+0000 7f7d599ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d541a0590 con 0x7f7d54074750 2023-07-20T18:58:49.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d52ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d54073170 0x7f7d54198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:49.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d537fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54108630 0x7f7d5419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:49.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d52ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d54073170 0x7f7d54198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54944/0 (socket says 172.21.15.57:54944) 2023-07-20T18:58:49.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d52ffd700 1 -- 172.21.15.57:0/1457324309 learned_addr learned my addr 172.21.15.57:0/1457324309 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:49.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d537fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54108630 0x7f7d5419fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:38264/0 (socket says 172.21.15.57:38264) 2023-07-20T18:58:49.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d527fc700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d54074750 0x7f7d54198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:49.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d52ffd700 1 -- 172.21.15.57:0/1457324309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d54074750 msgr2=0x7f7d54198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:49.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d52ffd700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d54074750 0x7f7d54198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d52ffd700 1 -- 172.21.15.57:0/1457324309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54108630 msgr2=0x7f7d5419fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:49.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d52ffd700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54108630 0x7f7d5419fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d52ffd700 1 -- 172.21.15.57:0/1457324309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440095f0 con 0x7f7d54073170 2023-07-20T18:58:49.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d537fe700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54108630 0x7f7d5419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:49.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.593+0000 7f7d52ffd700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d54073170 0x7f7d54198740 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d44004730 tx=0x7f7d44004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:49.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.594+0000 7f7d589fd700 1 -- 172.21.15.57:0/1457324309 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4401d070 con 0x7f7d54073170 2023-07-20T18:58:49.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.594+0000 7f7d589fd700 1 -- 172.21.15.57:0/1457324309 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f7d44004cb0 con 0x7f7d54073170 2023-07-20T18:58:49.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.594+0000 7f7d589fd700 1 -- 172.21.15.57:0/1457324309 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44005460 con 0x7f7d54073170 2023-07-20T18:58:49.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.594+0000 7f7d599ff700 1 -- 172.21.15.57:0/1457324309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d541a0810 con 0x7f7d54073170 2023-07-20T18:58:49.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.594+0000 7f7d599ff700 1 -- 172.21.15.57:0/1457324309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d541a0b80 con 0x7f7d54073170 2023-07-20T18:58:49.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.595+0000 7f7d599ff700 1 -- 172.21.15.57:0/1457324309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d54106d30 con 0x7f7d54073170 2023-07-20T18:58:49.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.597+0000 7f7d589fd700 1 -- 172.21.15.57:0/1457324309 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7d4400fb40 con 0x7f7d54073170 2023-07-20T18:58:49.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.598+0000 7f7d589fd700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7d3c0a11d0 0x7f7d3c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:49.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.598+0000 7f7d589fd700 1 -- 172.21.15.57:0/1457324309 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7d440cdcd0 con 0x7f7d54073170 2023-07-20T18:58:49.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.598+0000 7f7d527fc700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7d3c0a11d0 0x7f7d3c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:49.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.599+0000 7f7d527fc700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7d3c0a11d0 0x7f7d3c0a3690 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009c60 tx=0x7f7d48009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:49.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.602+0000 7f7d589fd700 1 -- 172.21.15.57:0/1457324309 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7d4405a780 con 0x7f7d54073170 2023-07-20T18:58:49.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.810+0000 7f7d599ff700 1 -- 172.21.15.57:0/1457324309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d540613b0 con 0x7f7d54073170 2023-07-20T18:58:49.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.810+0000 7f7d589fd700 1 -- 172.21.15.57:0/1457324309 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d44097260 con 0x7f7d54073170 2023-07-20T18:58:49.861 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-07-20T18:58:49.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 -- 172.21.15.57:0/1457324309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7d3c0a11d0 msgr2=0x7f7d3c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:49.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7d3c0a11d0 0x7f7d3c0a3690 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009c60 tx=0x7f7d48009260 comp rx=0 tx=0).stop 2023-07-20T18:58:49.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 -- 172.21.15.57:0/1457324309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d54073170 msgr2=0x7f7d54198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:49.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d54073170 0x7f7d54198740 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d44004730 tx=0x7f7d44004760 comp rx=0 tx=0).stop 2023-07-20T18:58:49.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 -- 172.21.15.57:0/1457324309 shutdown_connections 2023-07-20T18:58:49.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f7d3c0a11d0 0x7f7d3c0a3690 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54108630 0x7f7d5419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d54073170 0x7f7d54198740 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 --2- 172.21.15.57:0/1457324309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d54074750 0x7f7d54198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:49.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 -- 172.21.15.57:0/1457324309 >> 172.21.15.57:0/1457324309 conn(0x7f7d540fbcb0 msgr2=0x7f7d540fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:49.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 -- 172.21.15.57:0/1457324309 shutdown_connections 2023-07-20T18:58:49.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:49.813+0000 7f7d599ff700 1 -- 172.21.15.57:0/1457324309 wait complete. 2023-07-20T18:58:49.881 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-20T18:58:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.037+0000 7f4f3fbff700 1 -- 172.21.15.57:0/666862920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f38100760 msgr2=0x7f4f38100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.037+0000 7f4f3fbff700 1 --2- 172.21.15.57:0/666862920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f38100760 0x7f4f38100b80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4f340099c0 tx=0x7f4f34009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.037+0000 7f4f3fbff700 1 -- 172.21.15.57:0/666862920 shutdown_connections 2023-07-20T18:58:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.037+0000 7f4f3fbff700 1 --2- 172.21.15.57:0/666862920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f381022f0 0x7f4f38108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.037+0000 7f4f3fbff700 1 --2- 172.21.15.57:0/666862920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f38100760 0x7f4f38100b80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.037+0000 7f4f3fbff700 1 --2- 172.21.15.57:0/666862920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f38101950 0x7f4f38101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.037+0000 7f4f3fbff700 1 -- 172.21.15.57:0/666862920 >> 172.21.15.57:0/666862920 conn(0x7f4f380fbcb0 msgr2=0x7f4f380fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.038+0000 7f4f3fbff700 1 -- 172.21.15.57:0/666862920 shutdown_connections 2023-07-20T18:58:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.038+0000 7f4f3fbff700 1 -- 172.21.15.57:0/666862920 wait complete. 2023-07-20T18:58:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.038+0000 7f4f3fbff700 1 Processor -- start 2023-07-20T18:58:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3fbff700 1 -- start start 2023-07-20T18:58:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3fbff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f38100760 0x7f4f38198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:50.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3fbff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f38101950 0x7f4f38198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:50.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3fbff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f381022f0 0x7f4f3819fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:50.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3fbff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f381a0310 con 0x7f4f381022f0 2023-07-20T18:58:50.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3fbff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f381a0450 con 0x7f4f38100760 2023-07-20T18:58:50.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3fbff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f381a05e0 con 0x7f4f38101950 2023-07-20T18:58:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3e19c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f381022f0 0x7f4f3819fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3d19a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f38101950 0x7f4f38198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3d19a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f38101950 0x7f4f38198cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.57:54032/0 (socket says 172.21.15.57:54032) 2023-07-20T18:58:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3e19c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f381022f0 0x7f4f3819fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54972/0 (socket says 172.21.15.57:54972) 2023-07-20T18:58:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3d19a700 1 -- 172.21.15.57:0/413263547 learned_addr learned my addr 172.21.15.57:0/413263547 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:50.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.039+0000 7f4f3d99b700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f38100760 0x7f4f38198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:50.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f3e19c700 1 -- 172.21.15.57:0/413263547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f38101950 msgr2=0x7f4f38198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:50.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f3e19c700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f38101950 0x7f4f38198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f3e19c700 1 -- 172.21.15.57:0/413263547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f38100760 msgr2=0x7f4f38198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:50.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f3e19c700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f38100760 0x7f4f38198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f3e19c700 1 -- 172.21.15.57:0/413263547 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f340095f0 con 0x7f4f381022f0 2023-07-20T18:58:50.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f3e19c700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f381022f0 0x7f4f3819fd40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4f30007f10 tx=0x7f4f3000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:50.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f2effd700 1 -- 172.21.15.57:0/413263547 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30004db0 con 0x7f4f381022f0 2023-07-20T18:58:50.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f2effd700 1 -- 172.21.15.57:0/413263547 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f4f3000f040 con 0x7f4f381022f0 2023-07-20T18:58:50.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f3fbff700 1 -- 172.21.15.57:0/413263547 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f381a08c0 con 0x7f4f381022f0 2023-07-20T18:58:50.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f2effd700 1 -- 172.21.15.57:0/413263547 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30005a90 con 0x7f4f381022f0 2023-07-20T18:58:50.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.040+0000 7f4f3fbff700 1 -- 172.21.15.57:0/413263547 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f381a0d80 con 0x7f4f381022f0 2023-07-20T18:58:50.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.041+0000 7f4f3fbff700 1 -- 172.21.15.57:0/413263547 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f38192ef0 con 0x7f4f381022f0 2023-07-20T18:58:50.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.043+0000 7f4f2effd700 1 -- 172.21.15.57:0/413263547 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4f300050b0 con 0x7f4f381022f0 2023-07-20T18:58:50.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.046+0000 7f4f2effd700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4f240a1150 0x7f4f240a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:50.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.046+0000 7f4f2effd700 1 -- 172.21.15.57:0/413263547 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4f300cc430 con 0x7f4f381022f0 2023-07-20T18:58:50.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.046+0000 7f4f3d99b700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4f240a1150 0x7f4f240a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:50.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.047+0000 7f4f3d99b700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4f240a1150 0x7f4f240a3610 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009f50 tx=0x7f4f34003680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:50.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.050+0000 7f4f2effd700 1 -- 172.21.15.57:0/413263547 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4f3004e780 con 0x7f4f381022f0 2023-07-20T18:58:50.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:49 smithi057 ceph-mon[126340]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:50.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:49 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1668674242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:58:50.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:49 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/882233511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T18:58:50.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:49 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/1457324309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T18:58:50.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.259+0000 7f4f3fbff700 1 -- 172.21.15.57:0/413263547 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4f38061380 con 0x7f4f381022f0 2023-07-20T18:58:50.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.259+0000 7f4f2effd700 1 -- 172.21.15.57:0/413263547 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v86) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4f30095930 con 0x7f4f381022f0 2023-07-20T18:58:50.310 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-07-20T18:58:50.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 -- 172.21.15.57:0/413263547 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4f240a1150 msgr2=0x7f4f240a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:50.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4f240a1150 0x7f4f240a3610 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009f50 tx=0x7f4f34003680 comp rx=0 tx=0).stop 2023-07-20T18:58:50.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 -- 172.21.15.57:0/413263547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f381022f0 msgr2=0x7f4f3819fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:50.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f381022f0 0x7f4f3819fd40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4f30007f10 tx=0x7f4f3000d3b0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 -- 172.21.15.57:0/413263547 shutdown_connections 2023-07-20T18:58:50.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4f240a1150 0x7f4f240a3610 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f381022f0 0x7f4f3819fd40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f38100760 0x7f4f38198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 --2- 172.21.15.57:0/413263547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f38101950 0x7f4f38198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 -- 172.21.15.57:0/413263547 >> 172.21.15.57:0/413263547 conn(0x7f4f380fbcb0 msgr2=0x7f4f38107250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:50.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 -- 172.21.15.57:0/413263547 shutdown_connections 2023-07-20T18:58:50.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.262+0000 7f4f3fbff700 1 -- 172.21.15.57:0/413263547 wait complete. 2023-07-20T18:58:50.331 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-20T18:58:50.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:49 smithi192 ceph-mon[129200]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:50.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:49 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1668674242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:58:50.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:49 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/882233511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T18:58:50.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:49 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/1457324309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T18:58:50.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:49 smithi088 ceph-mon[128823]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T18:58:50.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:49 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1668674242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T18:58:50.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:49 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/882233511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T18:58:50.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:49 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/1457324309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T18:58:50.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.485+0000 7f1f07bff700 1 -- 172.21.15.57:0/78864559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f00101950 msgr2=0x7f1f00101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:50.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.485+0000 7f1f07bff700 1 --2- 172.21.15.57:0/78864559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f00101950 0x7f1f00101db0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009920 tx=0x7f1ef0009c30 comp rx=0 tx=0).stop 2023-07-20T18:58:50.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.485+0000 7f1f07bff700 1 -- 172.21.15.57:0/78864559 shutdown_connections 2023-07-20T18:58:50.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.485+0000 7f1f07bff700 1 --2- 172.21.15.57:0/78864559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f00100760 0x7f1f00100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.485+0000 7f1f07bff700 1 --2- 172.21.15.57:0/78864559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f001022f0 0x7f1f00108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.485+0000 7f1f07bff700 1 --2- 172.21.15.57:0/78864559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f00101950 0x7f1f00101db0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.485+0000 7f1f07bff700 1 -- 172.21.15.57:0/78864559 >> 172.21.15.57:0/78864559 conn(0x7f1f000fbcb0 msgr2=0x7f1f000fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:50.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.485+0000 7f1f07bff700 1 -- 172.21.15.57:0/78864559 shutdown_connections 2023-07-20T18:58:50.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.485+0000 7f1f07bff700 1 -- 172.21.15.57:0/78864559 wait complete. 2023-07-20T18:58:50.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.486+0000 7f1f07bff700 1 Processor -- start 2023-07-20T18:58:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.486+0000 7f1f07bff700 1 -- start start 2023-07-20T18:58:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f07bff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f00100760 0x7f1f001987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f07bff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f00101950 0x7f1f00198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f07bff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f001022f0 0x7f1f0019fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f07bff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f001a0350 con 0x7f1f00100760 2023-07-20T18:58:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f07bff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f001a0490 con 0x7f1f001022f0 2023-07-20T18:58:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f07bff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f001a0620 con 0x7f1f00101950 2023-07-20T18:58:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f0599b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f00100760 0x7f1f001987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f0599b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f00100760 0x7f1f001987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55002/0 (socket says 172.21.15.57:55002) 2023-07-20T18:58:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f0599b700 1 -- 172.21.15.57:0/2006801353 learned_addr learned my addr 172.21.15.57:0/2006801353 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f0519a700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f00101950 0x7f1f00198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f0619c700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f001022f0 0x7f1f0019fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f0599b700 1 -- 172.21.15.57:0/2006801353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f00101950 msgr2=0x7f1f00198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f0599b700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f00101950 0x7f1f00198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f0599b700 1 -- 172.21.15.57:0/2006801353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f001022f0 msgr2=0x7f1f0019fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f0599b700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f001022f0 0x7f1f0019fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.487+0000 7f1f0599b700 1 -- 172.21.15.57:0/2006801353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc0095f0 con 0x7f1f00100760 2023-07-20T18:58:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.488+0000 7f1f0599b700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f00100760 0x7f1f001987d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00ebf0 tx=0x7f1efc00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.488+0000 7f1ef6ffd700 1 -- 172.21.15.57:0/2006801353 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc009a10 con 0x7f1f00100760 2023-07-20T18:58:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.488+0000 7f1ef6ffd700 1 -- 172.21.15.57:0/2006801353 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f1efc00cd40 con 0x7f1f00100760 2023-07-20T18:58:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.488+0000 7f1f07bff700 1 -- 172.21.15.57:0/2006801353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef0009520 con 0x7f1f00100760 2023-07-20T18:58:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.488+0000 7f1ef6ffd700 1 -- 172.21.15.57:0/2006801353 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc018880 con 0x7f1f00100760 2023-07-20T18:58:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.488+0000 7f1f07bff700 1 -- 172.21.15.57:0/2006801353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f001a0c80 con 0x7f1f00100760 2023-07-20T18:58:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.489+0000 7f1f07bff700 1 -- 172.21.15.57:0/2006801353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f00192ea0 con 0x7f1f00100760 2023-07-20T18:58:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.491+0000 7f1ef6ffd700 1 -- 172.21.15.57:0/2006801353 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1efc010a80 con 0x7f1f00100760 2023-07-20T18:58:50.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.493+0000 7f1ef6ffd700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f1eec0a1150 0x7f1eec0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:50.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.493+0000 7f1ef6ffd700 1 -- 172.21.15.57:0/2006801353 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1efc014070 con 0x7f1f00100760 2023-07-20T18:58:50.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.493+0000 7f1f0519a700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f1eec0a1150 0x7f1eec0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:50.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.494+0000 7f1f0519a700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f1eec0a1150 0x7f1eec0a3610 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000fb90 tx=0x7f1ef0019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:50.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.494+0000 7f1ef6ffd700 1 -- 172.21.15.57:0/2006801353 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1efc096be0 con 0x7f1f00100760 2023-07-20T18:58:50.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.718+0000 7f1f07bff700 1 -- 172.21.15.57:0/2006801353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1f0002cc00 con 0x7f1f00100760 2023-07-20T18:58:50.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.718+0000 7f1ef6ffd700 1 -- 172.21.15.57:0/2006801353 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) v1 ==== 81+0+589 (secure 0 0 0) 0x7f1efc09a200 con 0x7f1f00100760 2023-07-20T18:58:50.769 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/2188156324 2023-07-21T18:52:02.610729+0000 2023-07-20T18:58:50.769 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3804015220 2023-07-21T18:52:02.610729+0000 2023-07-20T18:58:50.769 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/195678005 2023-07-21T18:51:34.614904+0000 2023-07-20T18:58:50.769 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/1582727629 2023-07-21T18:51:34.614904+0000 2023-07-20T18:58:50.770 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/2188156324 2023-07-21T18:52:02.610729+0000 2023-07-20T18:58:50.770 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/2044626900 2023-07-21T18:51:34.614904+0000 2023-07-20T18:58:50.770 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/748511752 2023-07-21T18:51:34.614904+0000 2023-07-20T18:58:50.770 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/1151100650 2023-07-21T18:52:02.610729+0000 2023-07-20T18:58:50.770 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/195678005 2023-07-21T18:51:34.614904+0000 2023-07-20T18:58:50.770 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3446341201 2023-07-21T18:52:02.610729+0000 2023-07-20T18:58:50.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 -- 172.21.15.57:0/2006801353 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f1eec0a1150 msgr2=0x7f1eec0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:50.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f1eec0a1150 0x7f1eec0a3610 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000fb90 tx=0x7f1ef0019040 comp rx=0 tx=0).stop 2023-07-20T18:58:50.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 -- 172.21.15.57:0/2006801353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f00100760 msgr2=0x7f1f001987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f00100760 0x7f1f001987d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00ebf0 tx=0x7f1efc00ef00 comp rx=0 tx=0).stop 2023-07-20T18:58:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 -- 172.21.15.57:0/2006801353 shutdown_connections 2023-07-20T18:58:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f1eec0a1150 0x7f1eec0a3610 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f001022f0 0x7f1f0019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f00100760 0x7f1f001987d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 --2- 172.21.15.57:0/2006801353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f00101950 0x7f1f00198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 -- 172.21.15.57:0/2006801353 >> 172.21.15.57:0/2006801353 conn(0x7f1f000fbcb0 msgr2=0x7f1f001072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:50.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 -- 172.21.15.57:0/2006801353 shutdown_connections 2023-07-20T18:58:50.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:50.721+0000 7f1f07bff700 1 -- 172.21.15.57:0/2006801353 wait complete. 2023-07-20T18:58:50.773 INFO:teuthology.orchestra.run.smithi057.stderr:listed 10 entries 2023-07-20T18:58:50.789 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-07-20T18:58:50.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:50.790 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-07-20T18:58:50.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:ceph-fuse[163650]: fuse finished with error 0 and tester_r 0 2023-07-20T18:58:50.861 INFO:teuthology.orchestra.run:waiting for 300 2023-07-20T18:58:51.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:50 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/413263547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T18:58:51.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:50 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2006801353' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T18:58:51.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:50 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/413263547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T18:58:51.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:50 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2006801353' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T18:58:51.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:50 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/413263547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T18:58:51.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:50 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2006801353' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T18:58:52.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:51 smithi057 ceph-mon[126340]: pgmap v280: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-20T18:58:52.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:51 smithi192 ceph-mon[129200]: pgmap v280: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-20T18:58:52.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:51 smithi088 ceph-mon[128823]: pgmap v280: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-20T18:58:54.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:53 smithi057 ceph-mon[126340]: pgmap v281: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-20T18:58:54.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:53 smithi192 ceph-mon[129200]: pgmap v281: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-20T18:58:54.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:53 smithi088 ceph-mon[128823]: pgmap v281: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-20T18:58:56.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:55 smithi057 ceph-mon[126340]: pgmap v282: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T18:58:56.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:55 smithi192 ceph-mon[129200]: pgmap v282: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T18:58:56.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:55 smithi088 ceph-mon[128823]: pgmap v282: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T18:58:56.963 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-07-20T18:58:56.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T18:58:56.963 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-07-20T18:58:57.020 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T18:58:57.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.180+0000 7f51939ff700 1 -- 172.21.15.57:0/999460449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f518c103b70 msgr2=0x7f518c103fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:57.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.180+0000 7f51939ff700 1 --2- 172.21.15.57:0/999460449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f518c103b70 0x7f518c103fd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f517c0099c0 tx=0x7f517c009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.180+0000 7f51939ff700 1 -- 172.21.15.57:0/999460449 shutdown_connections 2023-07-20T18:58:57.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.180+0000 7f51939ff700 1 --2- 172.21.15.57:0/999460449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f518c104510 0x7f518c10adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.180+0000 7f51939ff700 1 --2- 172.21.15.57:0/999460449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c102980 0x7f518c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.180+0000 7f51939ff700 1 --2- 172.21.15.57:0/999460449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f518c103b70 0x7f518c103fd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.180+0000 7f51939ff700 1 -- 172.21.15.57:0/999460449 >> 172.21.15.57:0/999460449 conn(0x7f518c0fdef0 msgr2=0x7f518c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:57.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.180+0000 7f51939ff700 1 -- 172.21.15.57:0/999460449 shutdown_connections 2023-07-20T18:58:57.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.180+0000 7f51939ff700 1 -- 172.21.15.57:0/999460449 wait complete. 2023-07-20T18:58:57.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.181+0000 7f51939ff700 1 Processor -- start 2023-07-20T18:58:57.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.181+0000 7f51939ff700 1 -- start start 2023-07-20T18:58:57.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.181+0000 7f51939ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f518c102980 0x7f518c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:57.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.181+0000 7f51939ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c103b70 0x7f518c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:57.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.181+0000 7f51939ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f518c104510 0x7f518c19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:57.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.181+0000 7f51939ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c1a0290 con 0x7f518c102980 2023-07-20T18:58:57.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.181+0000 7f51939ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c1a03d0 con 0x7f518c103b70 2023-07-20T18:58:57.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.181+0000 7f51939ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c1a0560 con 0x7f518c104510 2023-07-20T18:58:57.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f519179b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f518c102980 0x7f518c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:57.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f519179b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f518c102980 0x7f518c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39960/0 (socket says 172.21.15.57:39960) 2023-07-20T18:58:57.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f5191f9c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f518c104510 0x7f518c19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:57.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f519179b700 1 -- 172.21.15.57:0/116405730 learned_addr learned my addr 172.21.15.57:0/116405730 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:57.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f5190f9a700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c103b70 0x7f518c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:57.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f519179b700 1 -- 172.21.15.57:0/116405730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f518c104510 msgr2=0x7f518c19fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:57.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f519179b700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f518c104510 0x7f518c19fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f519179b700 1 -- 172.21.15.57:0/116405730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c103b70 msgr2=0x7f518c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:57.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f519179b700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c103b70 0x7f518c198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f519179b700 1 -- 172.21.15.57:0/116405730 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c0095f0 con 0x7f518c102980 2023-07-20T18:58:57.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f519179b700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f518c102980 0x7f518c198710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f518800e970 tx=0x7f518800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:57.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.182+0000 7f51827fc700 1 -- 172.21.15.57:0/116405730 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188009790 con 0x7f518c102980 2023-07-20T18:58:57.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.183+0000 7f51827fc700 1 -- 172.21.15.57:0/116405730 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f518800ccc0 con 0x7f518c102980 2023-07-20T18:58:57.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.183+0000 7f51939ff700 1 -- 172.21.15.57:0/116405730 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518c1a0840 con 0x7f518c102980 2023-07-20T18:58:57.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.183+0000 7f51827fc700 1 -- 172.21.15.57:0/116405730 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188003d80 con 0x7f518c102980 2023-07-20T18:58:57.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.183+0000 7f51939ff700 1 -- 172.21.15.57:0/116405730 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518c1a0d50 con 0x7f518c102980 2023-07-20T18:58:57.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.184+0000 7f51939ff700 1 -- 172.21.15.57:0/116405730 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518c109860 con 0x7f518c102980 2023-07-20T18:58:57.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.185+0000 7f51827fc700 1 -- 172.21.15.57:0/116405730 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f518800c7e0 con 0x7f518c102980 2023-07-20T18:58:57.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.187+0000 7f51827fc700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f51780a11a0 0x7f51780a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:57.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.187+0000 7f51827fc700 1 -- 172.21.15.57:0/116405730 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5188014070 con 0x7f518c102980 2023-07-20T18:58:57.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.187+0000 7f5190f9a700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f51780a11a0 0x7f51780a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:57.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.188+0000 7f5190f9a700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f51780a11a0 0x7f51780a3660 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f517c000c00 tx=0x7f517c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:57.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.191+0000 7f51827fc700 1 -- 172.21.15.57:0/116405730 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5188096200 con 0x7f518c102980 2023-07-20T18:58:57.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.400+0000 7f51939ff700 1 -- 172.21.15.57:0/116405730 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f518c02c970 con 0x7f518c102980 2023-07-20T18:58:57.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.402+0000 7f51827fc700 1 -- 172.21.15.57:0/116405730 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23911 (secure 0 0 0) 0x7f5188099820 con 0x7f518c102980 2023-07-20T18:58:57.453 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:57.453 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":86,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","created":"2023-07-20T18:51:01.245641+0000","modified":"2023-07-20T18:58:35.383850+0000","last_up_change":"2023-07-20T18:57:44.823544+0000","last_in_change":"2023-07-20T18:57:27.428782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T18:54:23.215777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T18:58:28.447992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T18:58:29.772740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T18:58:33.395093+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"627a67ea-1366-4b3f-babd-067f75bad1e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6803","nonce":3023441607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6805","nonce":3023441607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6809","nonce":3023441607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6807","nonce":3023441607}]},"public_addr":"172.21.15.57:6803/3023441607","cluster_addr":"172.21.15.57:6805/3023441607","heartbeat_back_addr":"172.21.15.57:6809/3023441607","heartbeat_front_addr":"172.21.15.57:6807/3023441607","state":["exists","up"]},{"osd":1,"uuid":"adcdb236-334e-4478-98a8-7d0628492082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6801","nonce":1809528555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6803","nonce":1809528555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6807","nonce":1809528555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6805","nonce":1809528555}]},"public_addr":"172.21.15.88:6801/1809528555","cluster_addr":"172.21.15.88:6803/1809528555","heartbeat_back_addr":"172.21.15.88:6807/1809528555","heartbeat_front_addr":"172.21.15.88:6805/1809528555","state":["exists","up"]},{"osd":2,"uuid":"3cd56229-2067-433a-b9f9-f47b1b878b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6801","nonce":3891687382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6803","nonce":3891687382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6807","nonce":3891687382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6805","nonce":3891687382}]},"public_addr":"172.21.15.192:6801/3891687382","cluster_addr":"172.21.15.192:6803/3891687382","heartbeat_back_addr":"172.21.15.192:6807/3891687382","heartbeat_front_addr":"172.21.15.192:6805/3891687382","state":["exists","up"]},{"osd":3,"uuid":"aae250ab-e242-4ef8-9c3a-a4fa36102c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6811","nonce":2920717314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6813","nonce":2920717314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6817","nonce":2920717314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6815","nonce":2920717314}]},"public_addr":"172.21.15.57:6811/2920717314","cluster_addr":"172.21.15.57:6813/2920717314","heartbeat_back_addr":"172.21.15.57:6817/2920717314","heartbeat_front_addr":"172.21.15.57:6815/2920717314","state":["exists","up"]},{"osd":4,"uuid":"8c8d1dfa-1cff-4f0a-ae72-22bd522ed696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6809","nonce":3911525610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6811","nonce":3911525610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6815","nonce":3911525610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6813","nonce":3911525610}]},"public_addr":"172.21.15.88:6809/3911525610","cluster_addr":"172.21.15.88:6811/3911525610","heartbeat_back_addr":"172.21.15.88:6815/3911525610","heartbeat_front_addr":"172.21.15.88:6813/3911525610","state":["exists","up"]},{"osd":5,"uuid":"5a2a56ab-2af9-4176-ba4d-b2138f0e7c75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6809","nonce":905436397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6811","nonce":905436397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6815","nonce":905436397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6813","nonce":905436397}]},"public_addr":"172.21.15.192:6809/905436397","cluster_addr":"172.21.15.192:6811/905436397","heartbeat_back_addr":"172.21.15.192:6815/905436397","heartbeat_front_addr":"172.21.15.192:6813/905436397","state":["exists","up"]},{"osd":6,"uuid":"03eafdda-278c-443c-b376-677eeb50262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6819","nonce":333894423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6821","nonce":333894423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6825","nonce":333894423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6823","nonce":333894423}]},"public_addr":"172.21.15.57:6819/333894423","cluster_addr":"172.21.15.57:6821/333894423","heartbeat_back_addr":"172.21.15.57:6825/333894423","heartbeat_front_addr":"172.21.15.57:6823/333894423","state":["exists","up"]},{"osd":7,"uuid":"3f01254e-be73-41c8-b979-3b6a7534e2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6817","nonce":3031485551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6819","nonce":3031485551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6823","nonce":3031485551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6821","nonce":3031485551}]},"public_addr":"172.21.15.88:6817/3031485551","cluster_addr":"172.21.15.88:6819/3031485551","heartbeat_back_addr":"172.21.15.88:6823/3031485551","heartbeat_front_addr":"172.21.15.88:6821/3031485551","state":["exists","up"]},{"osd":8,"uuid":"624a150e-f47b-4a76-8785-c818dd65342e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6817","nonce":918056390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6819","nonce":918056390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6823","nonce":918056390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6821","nonce":918056390}]},"public_addr":"172.21.15.192:6817/918056390","cluster_addr":"172.21.15.192:6819/918056390","heartbeat_back_addr":"172.21.15.192:6823/918056390","heartbeat_front_addr":"172.21.15.192:6821/918056390","state":["exists","up"]},{"osd":9,"uuid":"6d9cb69d-ed7b-4009-9ec5-027becc577d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6827","nonce":1773340763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6829","nonce":1773340763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6833","nonce":1773340763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6831","nonce":1773340763}]},"public_addr":"172.21.15.57:6827/1773340763","cluster_addr":"172.21.15.57:6829/1773340763","heartbeat_back_addr":"172.21.15.57:6833/1773340763","heartbeat_front_addr":"172.21.15.57:6831/1773340763","state":["exists","up"]},{"osd":10,"uuid":"d5799219-0979-40b1-a596-acd37f37a72d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6825","nonce":2781975579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6827","nonce":2781975579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6831","nonce":2781975579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6829","nonce":2781975579}]},"public_addr":"172.21.15.88:6825/2781975579","cluster_addr":"172.21.15.88:6827/2781975579","heartbeat_back_addr":"172.21.15.88:6831/2781975579","heartbeat_front_addr":"172.21.15.88:6829/2781975579","state":["exists","up"]},{"osd":11,"uuid":"e77cac6c-107e-41f9-8efd-e1140c4e3bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6825","nonce":2864190031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6827","nonce":2864190031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6831","nonce":2864190031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6829","nonce":2864190031}]},"public_addr":"172.21.15.192:6825/2864190031","cluster_addr":"172.21.15.192:6827/2864190031","heartbeat_back_addr":"172.21.15.192:6831/2864190031","heartbeat_front_addr":"172.21.15.192:6829/2864190031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:53:38.180971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:00.253039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:20.776122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:48.631094+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:09.288231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:29.595802+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:55.089126+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:15.637849+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:35.985101+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:01.544767+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:22.351743+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:43.255157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6801/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/3804015220":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6800/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1582727629":"2023-07-21T18:51:34.614904+0000","172.21.15.57:6800/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/2044626900":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/748511752":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1151100650":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6801/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/3446341201":"2023-07-21T18:52:02.610729+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T18:58:57.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 -- 172.21.15.57:0/116405730 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f51780a11a0 msgr2=0x7f51780a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:57.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f51780a11a0 0x7f51780a3660 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f517c000c00 tx=0x7f517c005fb0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 -- 172.21.15.57:0/116405730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f518c102980 msgr2=0x7f518c198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:57.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f518c102980 0x7f518c198710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f518800e970 tx=0x7f518800ec80 comp rx=0 tx=0).stop 2023-07-20T18:58:57.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 -- 172.21.15.57:0/116405730 shutdown_connections 2023-07-20T18:58:57.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f51780a11a0 0x7f51780a3660 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f518c104510 0x7f518c19fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f518c102980 0x7f518c198710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 --2- 172.21.15.57:0/116405730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c103b70 0x7f518c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 -- 172.21.15.57:0/116405730 >> 172.21.15.57:0/116405730 conn(0x7f518c0fdef0 msgr2=0x7f518c100120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:57.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 -- 172.21.15.57:0/116405730 shutdown_connections 2023-07-20T18:58:57.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.404+0000 7f51939ff700 1 -- 172.21.15.57:0/116405730 wait complete. 2023-07-20T18:58:57.473 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T18:58:57.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.628+0000 7f2f555ff700 1 -- 172.21.15.57:0/2455942461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f50100770 msgr2=0x7f2f50100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:57.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.628+0000 7f2f555ff700 1 --2- 172.21.15.57:0/2455942461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f50100770 0x7f2f50100b90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2f400099c0 tx=0x7f2f40009cd0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.628+0000 7f2f555ff700 1 -- 172.21.15.57:0/2455942461 shutdown_connections 2023-07-20T18:58:57.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.628+0000 7f2f555ff700 1 --2- 172.21.15.57:0/2455942461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f50102300 0x7f2f50108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.628+0000 7f2f555ff700 1 --2- 172.21.15.57:0/2455942461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f50100770 0x7f2f50100b90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.628+0000 7f2f555ff700 1 --2- 172.21.15.57:0/2455942461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f50101960 0x7f2f50101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.628+0000 7f2f555ff700 1 -- 172.21.15.57:0/2455942461 >> 172.21.15.57:0/2455942461 conn(0x7f2f500fbca0 msgr2=0x7f2f500fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:57.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.629+0000 7f2f555ff700 1 -- 172.21.15.57:0/2455942461 shutdown_connections 2023-07-20T18:58:57.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.629+0000 7f2f555ff700 1 -- 172.21.15.57:0/2455942461 wait complete. 2023-07-20T18:58:57.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.630+0000 7f2f555ff700 1 Processor -- start 2023-07-20T18:58:57.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.630+0000 7f2f555ff700 1 -- start start 2023-07-20T18:58:57.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.630+0000 7f2f555ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f50100770 0x7f2f50198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:57.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.630+0000 7f2f555ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f50101960 0x7f2f50198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:57.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f555ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f50102300 0x7f2f5019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f555ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f501a0290 con 0x7f2f50101960 2023-07-20T18:58:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f555ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f501a03d0 con 0x7f2f50102300 2023-07-20T18:58:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f555ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f501a0560 con 0x7f2f50100770 2023-07-20T18:58:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4e7fc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f50101960 0x7f2f50198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4e7fc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f50101960 0x7f2f50198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39980/0 (socket says 172.21.15.57:39980) 2023-07-20T18:58:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4e7fc700 1 -- 172.21.15.57:0/2678192521 learned_addr learned my addr 172.21.15.57:0/2678192521 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4effd700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f50100770 0x7f2f50198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4f7fe700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f50102300 0x7f2f5019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4e7fc700 1 -- 172.21.15.57:0/2678192521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f50100770 msgr2=0x7f2f50198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4e7fc700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f50100770 0x7f2f50198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4e7fc700 1 -- 172.21.15.57:0/2678192521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f50102300 msgr2=0x7f2f5019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4e7fc700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f50102300 0x7f2f5019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4e7fc700 1 -- 172.21.15.57:0/2678192521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f400095f0 con 0x7f2f50101960 2023-07-20T18:58:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4effd700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f50100770 0x7f2f50198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:58:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.631+0000 7f2f4e7fc700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f50101960 0x7f2f50198c50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400e9a0 tx=0x7f2f4400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:57.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.632+0000 7f2f37fff700 1 -- 172.21.15.57:0/2678192521 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44009790 con 0x7f2f50101960 2023-07-20T18:58:57.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.632+0000 7f2f555ff700 1 -- 172.21.15.57:0/2678192521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f501a0840 con 0x7f2f50101960 2023-07-20T18:58:57.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.632+0000 7f2f37fff700 1 -- 172.21.15.57:0/2678192521 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f2f44010430 con 0x7f2f50101960 2023-07-20T18:58:57.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.632+0000 7f2f37fff700 1 -- 172.21.15.57:0/2678192521 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44018680 con 0x7f2f50101960 2023-07-20T18:58:57.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.632+0000 7f2f555ff700 1 -- 172.21.15.57:0/2678192521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f501a0d50 con 0x7f2f50101960 2023-07-20T18:58:57.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.633+0000 7f2f555ff700 1 -- 172.21.15.57:0/2678192521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5010c0c0 con 0x7f2f50101960 2023-07-20T18:58:57.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.635+0000 7f2f37fff700 1 -- 172.21.15.57:0/2678192521 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2f44010910 con 0x7f2f50101960 2023-07-20T18:58:57.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.636+0000 7f2f37fff700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2f380a11a0 0x7f2f380a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:57.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.636+0000 7f2f37fff700 1 -- 172.21.15.57:0/2678192521 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2f44014070 con 0x7f2f50101960 2023-07-20T18:58:57.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.637+0000 7f2f4effd700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2f380a11a0 0x7f2f380a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:57.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.637+0000 7f2f4effd700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2f380a11a0 0x7f2f380a3660 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000b580 tx=0x7f2f40005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:57.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.640+0000 7f2f37fff700 1 -- 172.21.15.57:0/2678192521 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2f44051780 con 0x7f2f50101960 2023-07-20T18:58:57.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.852+0000 7f2f555ff700 1 -- 172.21.15.57:0/2678192521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f501a1390 con 0x7f2f50101960 2023-07-20T18:58:57.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.853+0000 7f2f37fff700 1 -- 172.21.15.57:0/2678192521 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23911 (secure 0 0 0) 0x7f2f440962b0 con 0x7f2f50101960 2023-07-20T18:58:57.904 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-07-20T18:58:57.904 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":86,"fsid":"38a58c3c-272e-11ee-9b34-001a4aab830c","created":"2023-07-20T18:51:01.245641+0000","modified":"2023-07-20T18:58:35.383850+0000","last_up_change":"2023-07-20T18:57:44.823544+0000","last_in_change":"2023-07-20T18:57:27.428782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T18:54:23.215777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T18:58:28.447992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T18:58:29.772740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T18:58:33.395093+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"627a67ea-1366-4b3f-babd-067f75bad1e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6803","nonce":3023441607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6805","nonce":3023441607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6809","nonce":3023441607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3023441607},{"type":"v1","addr":"172.21.15.57:6807","nonce":3023441607}]},"public_addr":"172.21.15.57:6803/3023441607","cluster_addr":"172.21.15.57:6805/3023441607","heartbeat_back_addr":"172.21.15.57:6809/3023441607","heartbeat_front_addr":"172.21.15.57:6807/3023441607","state":["exists","up"]},{"osd":1,"uuid":"adcdb236-334e-4478-98a8-7d0628492082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6801","nonce":1809528555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6803","nonce":1809528555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6807","nonce":1809528555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1809528555},{"type":"v1","addr":"172.21.15.88:6805","nonce":1809528555}]},"public_addr":"172.21.15.88:6801/1809528555","cluster_addr":"172.21.15.88:6803/1809528555","heartbeat_back_addr":"172.21.15.88:6807/1809528555","heartbeat_front_addr":"172.21.15.88:6805/1809528555","state":["exists","up"]},{"osd":2,"uuid":"3cd56229-2067-433a-b9f9-f47b1b878b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6801","nonce":3891687382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6803","nonce":3891687382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6807","nonce":3891687382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3891687382},{"type":"v1","addr":"172.21.15.192:6805","nonce":3891687382}]},"public_addr":"172.21.15.192:6801/3891687382","cluster_addr":"172.21.15.192:6803/3891687382","heartbeat_back_addr":"172.21.15.192:6807/3891687382","heartbeat_front_addr":"172.21.15.192:6805/3891687382","state":["exists","up"]},{"osd":3,"uuid":"aae250ab-e242-4ef8-9c3a-a4fa36102c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6811","nonce":2920717314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6813","nonce":2920717314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6817","nonce":2920717314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":2920717314},{"type":"v1","addr":"172.21.15.57:6815","nonce":2920717314}]},"public_addr":"172.21.15.57:6811/2920717314","cluster_addr":"172.21.15.57:6813/2920717314","heartbeat_back_addr":"172.21.15.57:6817/2920717314","heartbeat_front_addr":"172.21.15.57:6815/2920717314","state":["exists","up"]},{"osd":4,"uuid":"8c8d1dfa-1cff-4f0a-ae72-22bd522ed696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6809","nonce":3911525610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6811","nonce":3911525610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6815","nonce":3911525610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3911525610},{"type":"v1","addr":"172.21.15.88:6813","nonce":3911525610}]},"public_addr":"172.21.15.88:6809/3911525610","cluster_addr":"172.21.15.88:6811/3911525610","heartbeat_back_addr":"172.21.15.88:6815/3911525610","heartbeat_front_addr":"172.21.15.88:6813/3911525610","state":["exists","up"]},{"osd":5,"uuid":"5a2a56ab-2af9-4176-ba4d-b2138f0e7c75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6809","nonce":905436397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6811","nonce":905436397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6815","nonce":905436397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":905436397},{"type":"v1","addr":"172.21.15.192:6813","nonce":905436397}]},"public_addr":"172.21.15.192:6809/905436397","cluster_addr":"172.21.15.192:6811/905436397","heartbeat_back_addr":"172.21.15.192:6815/905436397","heartbeat_front_addr":"172.21.15.192:6813/905436397","state":["exists","up"]},{"osd":6,"uuid":"03eafdda-278c-443c-b376-677eeb50262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6819","nonce":333894423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6821","nonce":333894423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6825","nonce":333894423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":333894423},{"type":"v1","addr":"172.21.15.57:6823","nonce":333894423}]},"public_addr":"172.21.15.57:6819/333894423","cluster_addr":"172.21.15.57:6821/333894423","heartbeat_back_addr":"172.21.15.57:6825/333894423","heartbeat_front_addr":"172.21.15.57:6823/333894423","state":["exists","up"]},{"osd":7,"uuid":"3f01254e-be73-41c8-b979-3b6a7534e2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6817","nonce":3031485551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6819","nonce":3031485551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6823","nonce":3031485551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3031485551},{"type":"v1","addr":"172.21.15.88:6821","nonce":3031485551}]},"public_addr":"172.21.15.88:6817/3031485551","cluster_addr":"172.21.15.88:6819/3031485551","heartbeat_back_addr":"172.21.15.88:6823/3031485551","heartbeat_front_addr":"172.21.15.88:6821/3031485551","state":["exists","up"]},{"osd":8,"uuid":"624a150e-f47b-4a76-8785-c818dd65342e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6817","nonce":918056390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6819","nonce":918056390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6823","nonce":918056390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":918056390},{"type":"v1","addr":"172.21.15.192:6821","nonce":918056390}]},"public_addr":"172.21.15.192:6817/918056390","cluster_addr":"172.21.15.192:6819/918056390","heartbeat_back_addr":"172.21.15.192:6823/918056390","heartbeat_front_addr":"172.21.15.192:6821/918056390","state":["exists","up"]},{"osd":9,"uuid":"6d9cb69d-ed7b-4009-9ec5-027becc577d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6827","nonce":1773340763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6829","nonce":1773340763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6833","nonce":1773340763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1773340763},{"type":"v1","addr":"172.21.15.57:6831","nonce":1773340763}]},"public_addr":"172.21.15.57:6827/1773340763","cluster_addr":"172.21.15.57:6829/1773340763","heartbeat_back_addr":"172.21.15.57:6833/1773340763","heartbeat_front_addr":"172.21.15.57:6831/1773340763","state":["exists","up"]},{"osd":10,"uuid":"d5799219-0979-40b1-a596-acd37f37a72d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6825","nonce":2781975579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6827","nonce":2781975579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6831","nonce":2781975579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2781975579},{"type":"v1","addr":"172.21.15.88:6829","nonce":2781975579}]},"public_addr":"172.21.15.88:6825/2781975579","cluster_addr":"172.21.15.88:6827/2781975579","heartbeat_back_addr":"172.21.15.88:6831/2781975579","heartbeat_front_addr":"172.21.15.88:6829/2781975579","state":["exists","up"]},{"osd":11,"uuid":"e77cac6c-107e-41f9-8efd-e1140c4e3bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6825","nonce":2864190031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6827","nonce":2864190031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6831","nonce":2864190031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2864190031},{"type":"v1","addr":"172.21.15.192:6829","nonce":2864190031}]},"public_addr":"172.21.15.192:6825/2864190031","cluster_addr":"172.21.15.192:6827/2864190031","heartbeat_back_addr":"172.21.15.192:6831/2864190031","heartbeat_front_addr":"172.21.15.192:6829/2864190031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:53:38.180971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:00.253039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:20.776122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:54:48.631094+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:09.288231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:29.595802+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:55:55.089126+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:15.637849+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:56:35.985101+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:01.544767+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:22.351743+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T18:57:43.255157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6801/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/3804015220":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6800/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1582727629":"2023-07-21T18:51:34.614904+0000","172.21.15.57:6800/2188156324":"2023-07-21T18:52:02.610729+0000","172.21.15.57:0/2044626900":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/748511752":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/1151100650":"2023-07-21T18:52:02.610729+0000","172.21.15.57:6801/195678005":"2023-07-21T18:51:34.614904+0000","172.21.15.57:0/3446341201":"2023-07-21T18:52:02.610729+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T18:58:57.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.855+0000 7f2f555ff700 1 -- 172.21.15.57:0/2678192521 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2f380a11a0 msgr2=0x7f2f380a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:57.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.855+0000 7f2f555ff700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2f380a11a0 0x7f2f380a3660 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000b580 tx=0x7f2f40005fb0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.855+0000 7f2f555ff700 1 -- 172.21.15.57:0/2678192521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f50101960 msgr2=0x7f2f50198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:57.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.855+0000 7f2f555ff700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f50101960 0x7f2f50198c50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400e9a0 tx=0x7f2f4400ecb0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.856+0000 7f2f555ff700 1 -- 172.21.15.57:0/2678192521 shutdown_connections 2023-07-20T18:58:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.856+0000 7f2f555ff700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f2f380a11a0 0x7f2f380a3660 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.856+0000 7f2f555ff700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f50100770 0x7f2f50198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.856+0000 7f2f555ff700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f50102300 0x7f2f5019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.856+0000 7f2f555ff700 1 --2- 172.21.15.57:0/2678192521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f50101960 0x7f2f50198c50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.856+0000 7f2f555ff700 1 -- 172.21.15.57:0/2678192521 >> 172.21.15.57:0/2678192521 conn(0x7f2f500fbca0 msgr2=0x7f2f500fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.856+0000 7f2f555ff700 1 -- 172.21.15.57:0/2678192521 shutdown_connections 2023-07-20T18:58:57.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:57.856+0000 7f2f555ff700 1 -- 172.21.15.57:0/2678192521 wait complete. 2023-07-20T18:58:57.926 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-20T18:58:58.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.082+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2753642364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c981022f0 msgr2=0x7f4c98108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.082+0000 7f4c9dfff700 1 --2- 172.21.15.57:0/2753642364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c981022f0 0x7f4c98108ba0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000b0d0 tx=0x7f4c9000b3e0 comp rx=0 tx=0).stop 2023-07-20T18:58:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.083+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2753642364 shutdown_connections 2023-07-20T18:58:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.083+0000 7f4c9dfff700 1 --2- 172.21.15.57:0/2753642364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c98100760 0x7f4c98100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.083+0000 7f4c9dfff700 1 --2- 172.21.15.57:0/2753642364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c981022f0 0x7f4c98108ba0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.083+0000 7f4c9dfff700 1 --2- 172.21.15.57:0/2753642364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c98101950 0x7f4c98101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.083+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2753642364 >> 172.21.15.57:0/2753642364 conn(0x7f4c980fbcb0 msgr2=0x7f4c980fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.083+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2753642364 shutdown_connections 2023-07-20T18:58:58.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.083+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2753642364 wait complete. 2023-07-20T18:58:58.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c9dfff700 1 Processor -- start 2023-07-20T18:58:58.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c9dfff700 1 -- start start 2023-07-20T18:58:58.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c9dfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c98100760 0x7f4c98074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:58.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c9dfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c98101950 0x7f4c98072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:58.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c9dfff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c981022f0 0x7f4c980730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:58.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c9dfff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c98073840 con 0x7f4c98101950 2023-07-20T18:58:58.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c9dfff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980739b0 con 0x7f4c98100760 2023-07-20T18:58:58.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c9dfff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981a0690 con 0x7f4c981022f0 2023-07-20T18:58:58.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c8ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c98101950 0x7f4c98072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:58.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c8ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c98101950 0x7f4c98072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39994/0 (socket says 172.21.15.57:39994) 2023-07-20T18:58:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c8ffff700 1 -- 172.21.15.57:0/2895783820 learned_addr learned my addr 172.21.15.57:0/2895783820 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.084+0000 7f4c977fe700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c98100760 0x7f4c98074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c97fff700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c981022f0 0x7f4c980730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c8ffff700 1 -- 172.21.15.57:0/2895783820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c981022f0 msgr2=0x7f4c980730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c8ffff700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c981022f0 0x7f4c980730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c8ffff700 1 -- 172.21.15.57:0/2895783820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c98100760 msgr2=0x7f4c98074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c8ffff700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c98100760 0x7f4c98074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c8ffff700 1 -- 172.21.15.57:0/2895783820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c90009d00 con 0x7f4c98101950 2023-07-20T18:58:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c8ffff700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c98101950 0x7f4c98072b60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000e9a0 tx=0x7f4c8000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c957fa700 1 -- 172.21.15.57:0/2895783820 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c80009790 con 0x7f4c98101950 2023-07-20T18:58:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2895783820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c981a0970 con 0x7f4c98101950 2023-07-20T18:58:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c957fa700 1 -- 172.21.15.57:0/2895783820 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f4c80010430 con 0x7f4c98101950 2023-07-20T18:58:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c957fa700 1 -- 172.21.15.57:0/2895783820 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c80018680 con 0x7f4c98101950 2023-07-20T18:58:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.085+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2895783820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c981a0e80 con 0x7f4c98101950 2023-07-20T18:58:58.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.087+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2895783820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9810c110 con 0x7f4c98101950 2023-07-20T18:58:58.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.088+0000 7f4c957fa700 1 -- 172.21.15.57:0/2895783820 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4c80010910 con 0x7f4c98101950 2023-07-20T18:58:58.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.090+0000 7f4c957fa700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c840a1150 0x7f4c840a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:58.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.090+0000 7f4c957fa700 1 -- 172.21.15.57:0/2895783820 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4c80014070 con 0x7f4c98101950 2023-07-20T18:58:58.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.090+0000 7f4c977fe700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c840a1150 0x7f4c840a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:58.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.091+0000 7f4c977fe700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c840a1150 0x7f4c840a3610 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4c981017b0 tx=0x7f4c88009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:58.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.094+0000 7f4c957fa700 1 -- 172.21.15.57:0/2895783820 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4c80051780 con 0x7f4c98101950 2023-07-20T18:58:58.235 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:57 smithi057 ceph-mon[126340]: pgmap v283: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T18:58:58.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:57 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/116405730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:58.236 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:57 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2678192521' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:58.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:57 smithi192 ceph-mon[129200]: pgmap v283: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T18:58:58.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:57 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/116405730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:58.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:57 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2678192521' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:58.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:57 smithi088 ceph-mon[128823]: pgmap v283: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T18:58:58.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:57 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/116405730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:58.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:57 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2678192521' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T18:58:58.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.353+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2895783820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4c9810c2d0 con 0x7f4c98101950 2023-07-20T18:58:59.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.949+0000 7f4c957fa700 1 -- 172.21.15.57:0/2895783820 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4c800962b0 con 0x7f4c98101950 2023-07-20T18:58:59.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2895783820 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c840a1150 msgr2=0x7f4c840a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:59.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c840a1150 0x7f4c840a3610 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4c981017b0 tx=0x7f4c88009260 comp rx=0 tx=0).stop 2023-07-20T18:58:59.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2895783820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c98101950 msgr2=0x7f4c98072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:59.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c98101950 0x7f4c98072b60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000e9a0 tx=0x7f4c8000ecb0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2895783820 shutdown_connections 2023-07-20T18:58:59.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f4c840a1150 0x7f4c840a3610 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c981022f0 0x7f4c980730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c98100760 0x7f4c98074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 --2- 172.21.15.57:0/2895783820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c98101950 0x7f4c98072b60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2895783820 >> 172.21.15.57:0/2895783820 conn(0x7f4c980fbcb0 msgr2=0x7f4c980fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:59.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2895783820 shutdown_connections 2023-07-20T18:58:59.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:58.952+0000 7f4c9dfff700 1 -- 172.21.15.57:0/2895783820 wait complete. 2023-07-20T18:58:59.023 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2023-07-20T18:58:59.023 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-07-20T18:58:59.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.183+0000 7fc65efff700 1 -- 172.21.15.57:0/2953826576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc658106440 msgr2=0x7fc6581088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:59.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.183+0000 7fc65efff700 1 --2- 172.21.15.57:0/2953826576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc658106440 0x7fc6581088b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc65000b0a0 tx=0x7fc65000b3b0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.183+0000 7fc65efff700 1 -- 172.21.15.57:0/2953826576 shutdown_connections 2023-07-20T18:58:59.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.183+0000 7fc65efff700 1 --2- 172.21.15.57:0/2953826576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc658106440 0x7fc6581088b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.183+0000 7fc65efff700 1 --2- 172.21.15.57:0/2953826576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6581011e0 0x7fc6581035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.183+0000 7fc65efff700 1 --2- 172.21.15.57:0/2953826576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc658103b10 0x7fc658105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.183+0000 7fc65efff700 1 -- 172.21.15.57:0/2953826576 >> 172.21.15.57:0/2953826576 conn(0x7fc6580fabd0 msgr2=0x7fc6580fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:58:59.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.183+0000 7fc65efff700 1 -- 172.21.15.57:0/2953826576 shutdown_connections 2023-07-20T18:58:59.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.183+0000 7fc65efff700 1 -- 172.21.15.57:0/2953826576 wait complete. 2023-07-20T18:58:59.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.184+0000 7fc65efff700 1 Processor -- start 2023-07-20T18:58:59.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.184+0000 7fc65efff700 1 -- start start 2023-07-20T18:58:59.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.184+0000 7fc65efff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6581011e0 0x7fc658198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:59.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.184+0000 7fc65efff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc658103b10 0x7fc658198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:59.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.184+0000 7fc65efff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc658106440 0x7fc65819fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:59.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.184+0000 7fc65efff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6581a03e0 con 0x7fc6581011e0 2023-07-20T18:58:59.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.184+0000 7fc65efff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6581a0520 con 0x7fc658103b10 2023-07-20T18:58:59.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.184+0000 7fc65efff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6581a06b0 con 0x7fc658106440 2023-07-20T18:58:59.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc65cd9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6581011e0 0x7fc658198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:59.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc657fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc658103b10 0x7fc658198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:59.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc65cd9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6581011e0 0x7fc658198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40012/0 (socket says 172.21.15.57:40012) 2023-07-20T18:58:59.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc65cd9b700 1 -- 172.21.15.57:0/714364670 learned_addr learned my addr 172.21.15.57:0/714364670 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:58:59.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc65d59c700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc658106440 0x7fc65819fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:59.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc65cd9b700 1 -- 172.21.15.57:0/714364670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc658106440 msgr2=0x7fc65819fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:59.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc65cd9b700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc658106440 0x7fc65819fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc65cd9b700 1 -- 172.21.15.57:0/714364670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc658103b10 msgr2=0x7fc658198da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:58:59.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc65cd9b700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc658103b10 0x7fc658198da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:58:59.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc65cd9b700 1 -- 172.21.15.57:0/714364670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc650009d00 con 0x7fc6581011e0 2023-07-20T18:58:59.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.185+0000 7fc65cd9b700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6581011e0 0x7fc658198860 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00d980 tx=0x7fc64c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:59.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.186+0000 7fc655ffb700 1 -- 172.21.15.57:0/714364670 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c004220 con 0x7fc6581011e0 2023-07-20T18:58:59.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.186+0000 7fc65efff700 1 -- 172.21.15.57:0/714364670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6581a0990 con 0x7fc6581011e0 2023-07-20T18:58:59.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.186+0000 7fc655ffb700 1 -- 172.21.15.57:0/714364670 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fc64c007640 con 0x7fc6581011e0 2023-07-20T18:58:59.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.186+0000 7fc655ffb700 1 -- 172.21.15.57:0/714364670 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c00f690 con 0x7fc6581011e0 2023-07-20T18:58:59.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.186+0000 7fc65efff700 1 -- 172.21.15.57:0/714364670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6581a0ea0 con 0x7fc6581011e0 2023-07-20T18:58:59.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.187+0000 7fc65efff700 1 -- 172.21.15.57:0/714364670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc658107fc0 con 0x7fc6581011e0 2023-07-20T18:58:59.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.189+0000 7fc655ffb700 1 -- 172.21.15.57:0/714364670 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc64c004410 con 0x7fc6581011e0 2023-07-20T18:58:59.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.190+0000 7fc655ffb700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc6480a1150 0x7fc6480a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:58:59.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.190+0000 7fc655ffb700 1 -- 172.21.15.57:0/714364670 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc64c0cc9c0 con 0x7fc6581011e0 2023-07-20T18:58:59.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.190+0000 7fc657fff700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc6480a1150 0x7fc6480a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:58:59.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.191+0000 7fc657fff700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc6480a1150 0x7fc6480a3610 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc644005950 tx=0x7fc64400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:58:59.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.194+0000 7fc655ffb700 1 -- 172.21.15.57:0/714364670 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc64c095ec0 con 0x7fc6581011e0 2023-07-20T18:58:59.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:58 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2895783820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T18:58:59.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:58 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2895783820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T18:58:59.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:58 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2895783820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T18:58:59.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.453+0000 7fc65efff700 1 -- 172.21.15.57:0/714364670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7fc658061420 con 0x7fc6581011e0 2023-07-20T18:59:00.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.956+0000 7fc655ffb700 1 -- 172.21.15.57:0/714364670 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc64c016020 con 0x7fc6581011e0 2023-07-20T18:59:00.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 -- 172.21.15.57:0/714364670 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc6480a1150 msgr2=0x7fc6480a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:00.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc6480a1150 0x7fc6480a3610 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc644005950 tx=0x7fc64400b410 comp rx=0 tx=0).stop 2023-07-20T18:59:00.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 -- 172.21.15.57:0/714364670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6581011e0 msgr2=0x7fc658198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:00.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6581011e0 0x7fc658198860 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00d980 tx=0x7fc64c00dc90 comp rx=0 tx=0).stop 2023-07-20T18:59:00.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 -- 172.21.15.57:0/714364670 shutdown_connections 2023-07-20T18:59:00.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fc6480a1150 0x7fc6480a3610 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:00.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc658106440 0x7fc65819fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:00.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6581011e0 0x7fc658198860 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:00.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 --2- 172.21.15.57:0/714364670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc658103b10 0x7fc658198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:00.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 -- 172.21.15.57:0/714364670 >> 172.21.15.57:0/714364670 conn(0x7fc6580fabd0 msgr2=0x7fc6580ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:00.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 -- 172.21.15.57:0/714364670 shutdown_connections 2023-07-20T18:59:00.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:58:59.959+0000 7fc65efff700 1 -- 172.21.15.57:0/714364670 wait complete. 2023-07-20T18:59:00.033 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-20T18:59:00.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.191+0000 7fd4dcfff700 1 -- 172.21.15.57:0/1442686057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4d8100760 msgr2=0x7fd4d8100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:00.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.191+0000 7fd4dcfff700 1 --2- 172.21.15.57:0/1442686057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4d8100760 0x7fd4d8100b80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80099c0 tx=0x7fd4c8009cd0 comp rx=0 tx=0).stop 2023-07-20T18:59:00.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.191+0000 7fd4dcfff700 1 -- 172.21.15.57:0/1442686057 shutdown_connections 2023-07-20T18:59:00.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.191+0000 7fd4dcfff700 1 --2- 172.21.15.57:0/1442686057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4d81022f0 0x7fd4d8108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:00.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.191+0000 7fd4dcfff700 1 --2- 172.21.15.57:0/1442686057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4d8100760 0x7fd4d8100b80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:00.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.191+0000 7fd4dcfff700 1 --2- 172.21.15.57:0/1442686057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4d8101950 0x7fd4d8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:00.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.192+0000 7fd4dcfff700 1 -- 172.21.15.57:0/1442686057 >> 172.21.15.57:0/1442686057 conn(0x7fd4d80fbcb0 msgr2=0x7fd4d80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:00.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.192+0000 7fd4dcfff700 1 -- 172.21.15.57:0/1442686057 shutdown_connections 2023-07-20T18:59:00.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.192+0000 7fd4dcfff700 1 -- 172.21.15.57:0/1442686057 wait complete. 2023-07-20T18:59:00.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.193+0000 7fd4dcfff700 1 Processor -- start 2023-07-20T18:59:00.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.193+0000 7fd4dcfff700 1 -- start start 2023-07-20T18:59:00.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4dcfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4d8100760 0x7fd4d81965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:00.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4dcfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4d8101950 0x7fd4d8196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:00.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4dcfff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4d81022f0 0x7fd4d819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:00.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4dcfff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d819e140 con 0x7fd4d8100760 2023-07-20T18:59:00.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4dcfff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d819e280 con 0x7fd4d8101950 2023-07-20T18:59:00.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4dcfff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d819e410 con 0x7fd4d81022f0 2023-07-20T18:59:00.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4d659c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4d8100760 0x7fd4d81965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:00.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4d659c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4d8100760 0x7fd4d81965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40024/0 (socket says 172.21.15.57:40024) 2023-07-20T18:59:00.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4d659c700 1 -- 172.21.15.57:0/2512903228 learned_addr learned my addr 172.21.15.57:0/2512903228 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:59:00.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4d6d9d700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4d81022f0 0x7fd4d819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:00.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4d5d9b700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4d8101950 0x7fd4d8196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:00.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4d659c700 1 -- 172.21.15.57:0/2512903228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4d81022f0 msgr2=0x7fd4d819db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:00.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4d659c700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4d81022f0 0x7fd4d819db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:00.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.194+0000 7fd4d659c700 1 -- 172.21.15.57:0/2512903228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4d8101950 msgr2=0x7fd4d8196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:00.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.195+0000 7fd4d659c700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4d8101950 0x7fd4d8196b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:00.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.195+0000 7fd4d659c700 1 -- 172.21.15.57:0/2512903228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d0009c40 con 0x7fd4d8100760 2023-07-20T18:59:00.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.195+0000 7fd4d5d9b700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4d8101950 0x7fd4d8196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T18:59:00.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.195+0000 7fd4d659c700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4d8100760 0x7fd4d81965c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80098e0 tx=0x7fd4c800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:00.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.195+0000 7fd4c77fe700 1 -- 172.21.15.57:0/2512903228 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c801d070 con 0x7fd4d8100760 2023-07-20T18:59:00.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.195+0000 7fd4c77fe700 1 -- 172.21.15.57:0/2512903228 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7fd4c8005530 con 0x7fd4d8100760 2023-07-20T18:59:00.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.195+0000 7fd4c77fe700 1 -- 172.21.15.57:0/2512903228 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8004ad0 con 0x7fd4d8100760 2023-07-20T18:59:00.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.195+0000 7fd4dcfff700 1 -- 172.21.15.57:0/2512903228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c80095f0 con 0x7fd4d8100760 2023-07-20T18:59:00.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.195+0000 7fd4dcfff700 1 -- 172.21.15.57:0/2512903228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d819e8d0 con 0x7fd4d8100760 2023-07-20T18:59:00.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.196+0000 7fd4dcfff700 1 -- 172.21.15.57:0/2512903228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d81076e0 con 0x7fd4d8100760 2023-07-20T18:59:00.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.198+0000 7fd4c77fe700 1 -- 172.21.15.57:0/2512903228 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd4c8004c90 con 0x7fd4d8100760 2023-07-20T18:59:00.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.200+0000 7fd4c77fe700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd4c00a11d0 0x7fd4c00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:00.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.200+0000 7fd4c77fe700 1 -- 172.21.15.57:0/2512903228 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd4c80cd8c0 con 0x7fd4d8100760 2023-07-20T18:59:00.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.200+0000 7fd4d5d9b700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd4c00a11d0 0x7fd4c00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:00.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.200+0000 7fd4d5d9b700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd4c00a11d0 0x7fd4c00a3690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009cb0 tx=0x7fd4cc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:00.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.203+0000 7fd4c77fe700 1 -- 172.21.15.57:0/2512903228 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd4c8096eb0 con 0x7fd4d8100760 2023-07-20T18:59:00.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:59 smithi192 ceph-mon[129200]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T18:59:00.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:59 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2895783820' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T18:59:00.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:59 smithi192 ceph-mon[129200]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:59:00.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:58:59 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/714364670' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T18:59:00.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:59 smithi088 ceph-mon[128823]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T18:59:00.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:59 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2895783820' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T18:59:00.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:59 smithi088 ceph-mon[128823]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:59:00.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:58:59 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/714364670' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T18:59:00.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:59 smithi057 ceph-mon[126340]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T18:59:00.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:59 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2895783820' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T18:59:00.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:59 smithi057 ceph-mon[126340]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:59:00.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:58:59 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/714364670' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T18:59:00.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.462+0000 7fd4dcfff700 1 -- 172.21.15.57:0/2512903228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fd4d802c970 con 0x7fd4d8100760 2023-07-20T18:59:01.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.971+0000 7fd4c77fe700 1 -- 172.21.15.57:0/2512903228 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7fd4c809a4d0 con 0x7fd4d8100760 2023-07-20T18:59:01.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.973+0000 7fd4dcfff700 1 -- 172.21.15.57:0/2512903228 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd4c00a11d0 msgr2=0x7fd4c00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:01.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.973+0000 7fd4dcfff700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd4c00a11d0 0x7fd4c00a3690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009cb0 tx=0x7fd4cc009260 comp rx=0 tx=0).stop 2023-07-20T18:59:01.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.973+0000 7fd4dcfff700 1 -- 172.21.15.57:0/2512903228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4d8100760 msgr2=0x7fd4d81965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:01.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.973+0000 7fd4dcfff700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4d8100760 0x7fd4d81965c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80098e0 tx=0x7fd4c800bfd0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.974+0000 7fd4dcfff700 1 -- 172.21.15.57:0/2512903228 shutdown_connections 2023-07-20T18:59:01.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.974+0000 7fd4dcfff700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7fd4c00a11d0 0x7fd4c00a3690 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.974+0000 7fd4dcfff700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4d81022f0 0x7fd4d819db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.974+0000 7fd4dcfff700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4d8100760 0x7fd4d81965c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.974+0000 7fd4dcfff700 1 --2- 172.21.15.57:0/2512903228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4d8101950 0x7fd4d8196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.974+0000 7fd4dcfff700 1 -- 172.21.15.57:0/2512903228 >> 172.21.15.57:0/2512903228 conn(0x7fd4d80fbcb0 msgr2=0x7fd4d80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:01.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.974+0000 7fd4dcfff700 1 -- 172.21.15.57:0/2512903228 shutdown_connections 2023-07-20T18:59:01.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:00.974+0000 7fd4dcfff700 1 -- 172.21.15.57:0/2512903228 wait complete. 2023-07-20T18:59:01.049 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-20T18:59:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.208+0000 7f56045ff700 1 -- 172.21.15.57:0/4141640638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55fc1022f0 msgr2=0x7f55fc108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.208+0000 7f56045ff700 1 --2- 172.21.15.57:0/4141640638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55fc1022f0 0x7f55fc108ba0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f55f400b0d0 tx=0x7f55f400b3e0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.208+0000 7f56045ff700 1 -- 172.21.15.57:0/4141640638 shutdown_connections 2023-07-20T18:59:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.208+0000 7f56045ff700 1 --2- 172.21.15.57:0/4141640638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc100760 0x7f55fc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.208+0000 7f56045ff700 1 --2- 172.21.15.57:0/4141640638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55fc1022f0 0x7f55fc108ba0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.208+0000 7f56045ff700 1 --2- 172.21.15.57:0/4141640638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55fc101950 0x7f55fc101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.208+0000 7f56045ff700 1 -- 172.21.15.57:0/4141640638 >> 172.21.15.57:0/4141640638 conn(0x7f55fc0fbcb0 msgr2=0x7f55fc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:01.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.208+0000 7f56045ff700 1 -- 172.21.15.57:0/4141640638 shutdown_connections 2023-07-20T18:59:01.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.208+0000 7f56045ff700 1 -- 172.21.15.57:0/4141640638 wait complete. 2023-07-20T18:59:01.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.209+0000 7f56045ff700 1 Processor -- start 2023-07-20T18:59:01.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.209+0000 7f56045ff700 1 -- start start 2023-07-20T18:59:01.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.209+0000 7f56045ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc100760 0x7f55fc074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:01.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f56045ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55fc101950 0x7f55fc072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:01.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f56045ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55fc1022f0 0x7f55fc0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f56045ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc073840 con 0x7f55fc101950 2023-07-20T18:59:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f56045ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc0739b0 con 0x7f55fc1022f0 2023-07-20T18:59:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f56045ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc1a0610 con 0x7f55fc100760 2023-07-20T18:59:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f5601b9a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55fc101950 0x7f55fc072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f5601b9a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55fc101950 0x7f55fc072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40040/0 (socket says 172.21.15.57:40040) 2023-07-20T18:59:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f5601b9a700 1 -- 172.21.15.57:0/3441015755 learned_addr learned my addr 172.21.15.57:0/3441015755 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:59:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f560239b700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc100760 0x7f55fc074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f5602b9c700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55fc1022f0 0x7f55fc0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f5601b9a700 1 -- 172.21.15.57:0/3441015755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc100760 msgr2=0x7f55fc074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f5601b9a700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc100760 0x7f55fc074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f5601b9a700 1 -- 172.21.15.57:0/3441015755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55fc1022f0 msgr2=0x7f55fc0730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f5601b9a700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55fc1022f0 0x7f55fc0730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f5601b9a700 1 -- 172.21.15.57:0/3441015755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f4009d00 con 0x7f55fc101950 2023-07-20T18:59:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.210+0000 7f560239b700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc100760 0x7f55fc074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T18:59:01.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.211+0000 7f5601b9a700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55fc101950 0x7f55fc072b60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00e9a0 tx=0x7f55ec00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:01.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.211+0000 7f55f37fe700 1 -- 172.21.15.57:0/3441015755 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec009790 con 0x7f55fc101950 2023-07-20T18:59:01.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.211+0000 7f55f37fe700 1 -- 172.21.15.57:0/3441015755 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f55ec010430 con 0x7f55fc101950 2023-07-20T18:59:01.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.211+0000 7f56045ff700 1 -- 172.21.15.57:0/3441015755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc1a08f0 con 0x7f55fc101950 2023-07-20T18:59:01.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.211+0000 7f55f37fe700 1 -- 172.21.15.57:0/3441015755 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec018680 con 0x7f55fc101950 2023-07-20T18:59:01.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.211+0000 7f56045ff700 1 -- 172.21.15.57:0/3441015755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc1a0e00 con 0x7f55fc101950 2023-07-20T18:59:01.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.212+0000 7f56045ff700 1 -- 172.21.15.57:0/3441015755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc107c30 con 0x7f55fc101950 2023-07-20T18:59:01.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.214+0000 7f55f37fe700 1 -- 172.21.15.57:0/3441015755 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f55ec010910 con 0x7f55fc101950 2023-07-20T18:59:01.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.215+0000 7f55f37fe700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f55e80a1150 0x7f55e80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:01.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.215+0000 7f55f37fe700 1 -- 172.21.15.57:0/3441015755 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f55ec014070 con 0x7f55fc101950 2023-07-20T18:59:01.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.216+0000 7f560239b700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f55e80a1150 0x7f55e80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:01.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.216+0000 7f560239b700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f55e80a1150 0x7f55e80a3610 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f55fc1017b0 tx=0x7f55f8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:01.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.219+0000 7f55f37fe700 1 -- 172.21.15.57:0/3441015755 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f55ec051780 con 0x7f55fc101950 2023-07-20T18:59:01.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:00 smithi192 ceph-mon[129200]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T18:59:01.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:00 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/714364670' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T18:59:01.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:00 smithi192 ceph-mon[129200]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T18:59:01.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:00 smithi192 ceph-mon[129200]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:59:01.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:00 smithi192 ceph-mon[129200]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-20T18:59:01.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:00 smithi192 ceph-mon[129200]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-20T18:59:01.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:00 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2512903228' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T18:59:01.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:00 smithi088 ceph-mon[128823]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T18:59:01.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:00 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/714364670' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T18:59:01.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:00 smithi088 ceph-mon[128823]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T18:59:01.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:00 smithi088 ceph-mon[128823]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:59:01.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:00 smithi088 ceph-mon[128823]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-20T18:59:01.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:00 smithi088 ceph-mon[128823]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-20T18:59:01.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:00 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2512903228' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T18:59:01.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:00 smithi057 ceph-mon[126340]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T18:59:01.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:00 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/714364670' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T18:59:01.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:00 smithi057 ceph-mon[126340]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T18:59:01.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:00 smithi057 ceph-mon[126340]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T18:59:01.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:00 smithi057 ceph-mon[126340]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-20T18:59:01.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:00 smithi057 ceph-mon[126340]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-20T18:59:01.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:00 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2512903228' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T18:59:01.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.475+0000 7f56045ff700 1 -- 172.21.15.57:0/3441015755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f55fc02c970 con 0x7f55fc101950 2023-07-20T18:59:02.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.981+0000 7f55f37fe700 1 -- 172.21.15.57:0/3441015755 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f55ec0962b0 con 0x7f55fc101950 2023-07-20T18:59:02.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.984+0000 7f56045ff700 1 -- 172.21.15.57:0/3441015755 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f55e80a1150 msgr2=0x7f55e80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:02.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.984+0000 7f56045ff700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f55e80a1150 0x7f55e80a3610 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f55fc1017b0 tx=0x7f55f8009260 comp rx=0 tx=0).stop 2023-07-20T18:59:02.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.984+0000 7f56045ff700 1 -- 172.21.15.57:0/3441015755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55fc101950 msgr2=0x7f55fc072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.984+0000 7f56045ff700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55fc101950 0x7f55fc072b60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00e9a0 tx=0x7f55ec00ecb0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.984+0000 7f56045ff700 1 -- 172.21.15.57:0/3441015755 shutdown_connections 2023-07-20T18:59:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.984+0000 7f56045ff700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f55e80a1150 0x7f55e80a3610 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.984+0000 7f56045ff700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc100760 0x7f55fc074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.984+0000 7f56045ff700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55fc1022f0 0x7f55fc0730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.984+0000 7f56045ff700 1 --2- 172.21.15.57:0/3441015755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55fc101950 0x7f55fc072b60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.984+0000 7f56045ff700 1 -- 172.21.15.57:0/3441015755 >> 172.21.15.57:0/3441015755 conn(0x7f55fc0fbcb0 msgr2=0x7f55fc104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:02.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.985+0000 7f56045ff700 1 -- 172.21.15.57:0/3441015755 shutdown_connections 2023-07-20T18:59:02.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:01.985+0000 7f56045ff700 1 -- 172.21.15.57:0/3441015755 wait complete. 2023-07-20T18:59:02.059 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2023-07-20T18:59:02.060 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2023-07-20T18:59:02.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.218+0000 7f76aa7ff700 1 -- 172.21.15.57:0/403955496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76a4100760 msgr2=0x7f76a4100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:02.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.218+0000 7f76aa7ff700 1 --2- 172.21.15.57:0/403955496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76a4100760 0x7f76a4100b80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f76940099c0 tx=0x7f7694009cd0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.218+0000 7f76aa7ff700 1 -- 172.21.15.57:0/403955496 shutdown_connections 2023-07-20T18:59:02.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.218+0000 7f76aa7ff700 1 --2- 172.21.15.57:0/403955496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76a41022f0 0x7f76a4108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.218+0000 7f76aa7ff700 1 --2- 172.21.15.57:0/403955496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76a4100760 0x7f76a4100b80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.218+0000 7f76aa7ff700 1 --2- 172.21.15.57:0/403955496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76a4101950 0x7f76a4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.218+0000 7f76aa7ff700 1 -- 172.21.15.57:0/403955496 >> 172.21.15.57:0/403955496 conn(0x7f76a40fbcf0 msgr2=0x7f76a40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:02.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.218+0000 7f76aa7ff700 1 -- 172.21.15.57:0/403955496 shutdown_connections 2023-07-20T18:59:02.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.218+0000 7f76aa7ff700 1 -- 172.21.15.57:0/403955496 wait complete. 2023-07-20T18:59:02.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.219+0000 7f76aa7ff700 1 Processor -- start 2023-07-20T18:59:02.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.219+0000 7f76aa7ff700 1 -- start start 2023-07-20T18:59:02.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.219+0000 7f76aa7ff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76a4100760 0x7f76a4074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:02.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.219+0000 7f76aa7ff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76a4101950 0x7f76a4072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:02.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.219+0000 7f76aa7ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76a41022f0 0x7f76a40730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:02.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.219+0000 7f76aa7ff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a4073840 con 0x7f76a4101950 2023-07-20T18:59:02.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.219+0000 7f76aa7ff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a40739b0 con 0x7f76a4100760 2023-07-20T18:59:02.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.219+0000 7f76aa7ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a41a05e0 con 0x7f76a41022f0 2023-07-20T18:59:02.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f769bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76a4101950 0x7f76a4072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:02.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f769bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76a4101950 0x7f76a4072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40072/0 (socket says 172.21.15.57:40072) 2023-07-20T18:59:02.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f769bfff700 1 -- 172.21.15.57:0/4095437197 learned_addr learned my addr 172.21.15.57:0/4095437197 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:59:02.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f76a3fff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76a4100760 0x7f76a4074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:02.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f76a8d9c700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76a41022f0 0x7f76a40730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:02.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f769bfff700 1 -- 172.21.15.57:0/4095437197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76a41022f0 msgr2=0x7f76a40730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:02.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f769bfff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76a41022f0 0x7f76a40730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f769bfff700 1 -- 172.21.15.57:0/4095437197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76a4100760 msgr2=0x7f76a4074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:02.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f769bfff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76a4100760 0x7f76a4074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:02.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f769bfff700 1 -- 172.21.15.57:0/4095437197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76940095f0 con 0x7f76a4101950 2023-07-20T18:59:02.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f769bfff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76a4101950 0x7f76a4072b60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f768c00e9a0 tx=0x7f768c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:02.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.220+0000 7f76a1ffb700 1 -- 172.21.15.57:0/4095437197 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c009790 con 0x7f76a4101950 2023-07-20T18:59:02.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.221+0000 7f76aa7ff700 1 -- 172.21.15.57:0/4095437197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a41a08c0 con 0x7f76a4101950 2023-07-20T18:59:02.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.221+0000 7f76a1ffb700 1 -- 172.21.15.57:0/4095437197 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f768c00cbd0 con 0x7f76a4101950 2023-07-20T18:59:02.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.221+0000 7f76a1ffb700 1 -- 172.21.15.57:0/4095437197 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c018680 con 0x7f76a4101950 2023-07-20T18:59:02.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.221+0000 7f76aa7ff700 1 -- 172.21.15.57:0/4095437197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a41a0dd0 con 0x7f76a4101950 2023-07-20T18:59:02.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.222+0000 7f76aa7ff700 1 -- 172.21.15.57:0/4095437197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a4107ca0 con 0x7f76a4101950 2023-07-20T18:59:02.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.224+0000 7f76a1ffb700 1 -- 172.21.15.57:0/4095437197 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f768c0188a0 con 0x7f76a4101950 2023-07-20T18:59:02.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.225+0000 7f76a1ffb700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f76900a1150 0x7f76900a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:02.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.225+0000 7f76a1ffb700 1 -- 172.21.15.57:0/4095437197 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f768c014070 con 0x7f76a4101950 2023-07-20T18:59:02.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.226+0000 7f76a3fff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f76900a1150 0x7f76900a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:02.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.226+0000 7f76a3fff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f76900a1150 0x7f76900a3610 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7694006010 tx=0x7f7694005c20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:02.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.229+0000 7f76a1ffb700 1 -- 172.21.15.57:0/4095437197 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f768c051780 con 0x7f76a4101950 2023-07-20T18:59:02.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:01 smithi192 ceph-mon[129200]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-07-20T18:59:02.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:01 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.192:6838/1456232015,v1:172.21.15.192:6839/1456232015] up:active 2023-07-20T18:59:02.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:01 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/2512903228' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T18:59:02.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:01 smithi192 ceph-mon[129200]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T18:59:02.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:01 smithi192 ceph-mon[129200]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-20T18:59:02.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:01 smithi192 ceph-mon[129200]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-20T18:59:02.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:01 smithi192 ceph-mon[129200]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-20T18:59:02.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:01 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3441015755' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T18:59:02.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:01 smithi088 ceph-mon[128823]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-07-20T18:59:02.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:01 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.192:6838/1456232015,v1:172.21.15.192:6839/1456232015] up:active 2023-07-20T18:59:02.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:01 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/2512903228' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T18:59:02.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:01 smithi088 ceph-mon[128823]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T18:59:02.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:01 smithi088 ceph-mon[128823]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-20T18:59:02.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:01 smithi088 ceph-mon[128823]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-20T18:59:02.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:01 smithi088 ceph-mon[128823]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-20T18:59:02.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:01 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3441015755' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T18:59:02.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:01 smithi057 ceph-mon[126340]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-07-20T18:59:02.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:01 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.192:6838/1456232015,v1:172.21.15.192:6839/1456232015] up:active 2023-07-20T18:59:02.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:01 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/2512903228' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T18:59:02.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:01 smithi057 ceph-mon[126340]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T18:59:02.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:01 smithi057 ceph-mon[126340]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-20T18:59:02.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:01 smithi057 ceph-mon[126340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-20T18:59:02.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:01 smithi057 ceph-mon[126340]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-20T18:59:02.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:01 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3441015755' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T18:59:02.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:02.475+0000 7f76aa7ff700 1 -- 172.21.15.57:0/4095437197 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f76a4060bb0 con 0x7f76900a1150 2023-07-20T18:59:02.986 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:59:02.481+0000 7f038fe5e700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T18:59:02.986 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:59:02.481+0000 7f038fe5e700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T18:59:02.986 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:59:02.481+0000 7f038fe5e700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T18:59:02.987 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:59:02.481+0000 7f038fe5e700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T18:59:02.987 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:59:02.481+0000 7f038fe5e700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T18:59:03.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.033+0000 7f76a1ffb700 1 -- 172.21.15.57:0/4095437197 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f76a4060bb0 con 0x7f76900a1150 2023-07-20T18:59:03.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 -- 172.21.15.57:0/4095437197 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f76900a1150 msgr2=0x7f76900a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:03.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f76900a1150 0x7f76900a3610 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7694006010 tx=0x7f7694005c20 comp rx=0 tx=0).stop 2023-07-20T18:59:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 -- 172.21.15.57:0/4095437197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76a4101950 msgr2=0x7f76a4072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76a4101950 0x7f76a4072b60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f768c00e9a0 tx=0x7f768c00ecb0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 -- 172.21.15.57:0/4095437197 shutdown_connections 2023-07-20T18:59:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f76900a1150 0x7f76900a3610 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76a41022f0 0x7f76a40730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76a4100760 0x7f76a4074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 --2- 172.21.15.57:0/4095437197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76a4101950 0x7f76a4072b60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 -- 172.21.15.57:0/4095437197 >> 172.21.15.57:0/4095437197 conn(0x7f76a40fbcf0 msgr2=0x7f76a4104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:03.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 -- 172.21.15.57:0/4095437197 shutdown_connections 2023-07-20T18:59:03.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.036+0000 7f76aa7ff700 1 -- 172.21.15.57:0/4095437197 wait complete. 2023-07-20T18:59:03.106 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa '' 2023-07-20T18:59:03.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.268+0000 7f45e2dff700 1 -- 172.21.15.57:0/962011704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45dc1022f0 msgr2=0x7f45dc108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:03.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.268+0000 7f45e2dff700 1 --2- 172.21.15.57:0/962011704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45dc1022f0 0x7f45dc108ba0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f45d400b0d0 tx=0x7f45d400b3e0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.268+0000 7f45e2dff700 1 -- 172.21.15.57:0/962011704 shutdown_connections 2023-07-20T18:59:03.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.268+0000 7f45e2dff700 1 --2- 172.21.15.57:0/962011704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc100760 0x7f45dc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.268+0000 7f45e2dff700 1 --2- 172.21.15.57:0/962011704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45dc1022f0 0x7f45dc108ba0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.268+0000 7f45e2dff700 1 --2- 172.21.15.57:0/962011704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc101950 0x7f45dc101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.268+0000 7f45e2dff700 1 -- 172.21.15.57:0/962011704 >> 172.21.15.57:0/962011704 conn(0x7f45dc0fbcb0 msgr2=0x7f45dc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:03.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.268+0000 7f45e2dff700 1 -- 172.21.15.57:0/962011704 shutdown_connections 2023-07-20T18:59:03.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.268+0000 7f45e2dff700 1 -- 172.21.15.57:0/962011704 wait complete. 2023-07-20T18:59:03.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.269+0000 7f45e2dff700 1 Processor -- start 2023-07-20T18:59:03.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.269+0000 7f45e2dff700 1 -- start start 2023-07-20T18:59:03.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.269+0000 7f45e2dff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc100760 0x7f45dc074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:03.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.269+0000 7f45e2dff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45dc101950 0x7f45dc072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:03.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.269+0000 7f45e2dff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc1022f0 0x7f45dc0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:03.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.269+0000 7f45e2dff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc073840 con 0x7f45dc101950 2023-07-20T18:59:03.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.269+0000 7f45e2dff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc0739b0 con 0x7f45dc1022f0 2023-07-20T18:59:03.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.269+0000 7f45e2dff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1a0660 con 0x7f45dc100760 2023-07-20T18:59:03.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45dbfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45dc101950 0x7f45dc072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:03.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45dbfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45dc101950 0x7f45dc072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40110/0 (socket says 172.21.15.57:40110) 2023-07-20T18:59:03.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45dbfff700 1 -- 172.21.15.57:0/1509368740 learned_addr learned my addr 172.21.15.57:0/1509368740 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:59:03.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45e139c700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc1022f0 0x7f45dc0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:03.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45e0b9b700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc100760 0x7f45dc074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:03.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45dbfff700 1 -- 172.21.15.57:0/1509368740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc100760 msgr2=0x7f45dc074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:03.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45dbfff700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc100760 0x7f45dc074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45dbfff700 1 -- 172.21.15.57:0/1509368740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc1022f0 msgr2=0x7f45dc0730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:03.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45dbfff700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc1022f0 0x7f45dc0730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45dbfff700 1 -- 172.21.15.57:0/1509368740 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d4009d00 con 0x7f45dc101950 2023-07-20T18:59:03.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.270+0000 7f45dbfff700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45dc101950 0x7f45dc072b60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f45c800e9a0 tx=0x7f45c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:03.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.271+0000 7f45d9ffb700 1 -- 172.21.15.57:0/1509368740 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8009790 con 0x7f45dc101950 2023-07-20T18:59:03.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.271+0000 7f45e2dff700 1 -- 172.21.15.57:0/1509368740 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc1a0940 con 0x7f45dc101950 2023-07-20T18:59:03.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.271+0000 7f45d9ffb700 1 -- 172.21.15.57:0/1509368740 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f45c8010430 con 0x7f45dc101950 2023-07-20T18:59:03.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.271+0000 7f45d9ffb700 1 -- 172.21.15.57:0/1509368740 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8018680 con 0x7f45dc101950 2023-07-20T18:59:03.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.271+0000 7f45e2dff700 1 -- 172.21.15.57:0/1509368740 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45dc1a0ea0 con 0x7f45dc101950 2023-07-20T18:59:03.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.272+0000 7f45e2dff700 1 -- 172.21.15.57:0/1509368740 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45dc1075a0 con 0x7f45dc101950 2023-07-20T18:59:03.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.274+0000 7f45d9ffb700 1 -- 172.21.15.57:0/1509368740 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f45c8010910 con 0x7f45dc101950 2023-07-20T18:59:03.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.275+0000 7f45d9ffb700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45cc0a11d0 0x7f45cc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:03.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.275+0000 7f45d9ffb700 1 -- 172.21.15.57:0/1509368740 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f45c8014070 con 0x7f45dc101950 2023-07-20T18:59:03.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.275+0000 7f45e0b9b700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45cc0a11d0 0x7f45cc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:03.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.276+0000 7f45e0b9b700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45cc0a11d0 0x7f45cc0a3690 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f45dc1017b0 tx=0x7f45d0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:03.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.279+0000 7f45d9ffb700 1 -- 172.21.15.57:0/1509368740 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f45c8051780 con 0x7f45dc101950 2023-07-20T18:59:03.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.88:6836/678288704,v1:172.21.15.88:6837/678288704] up:active 2023-07-20T18:59:03.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='client.? 172.21.15.57:0/3441015755' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T18:59:03.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T18:59:03.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-20T18:59:03.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-20T18:59:03.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-20T18:59:03.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-07-20T18:59:03.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2023-07-20T18:59:03.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2023-07-20T18:59:03.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-07-20T18:59:03.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2023-07-20T18:59:03.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-07-20T18:59:03.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]: dispatch 2023-07-20T18:59:03.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-07-20T18:59:03.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2023-07-20T18:59:03.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:02 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2023-07-20T18:59:03.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.88:6836/678288704,v1:172.21.15.88:6837/678288704] up:active 2023-07-20T18:59:03.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='client.? 172.21.15.57:0/3441015755' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T18:59:03.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T18:59:03.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-20T18:59:03.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-20T18:59:03.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-20T18:59:03.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-07-20T18:59:03.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2023-07-20T18:59:03.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2023-07-20T18:59:03.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-07-20T18:59:03.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2023-07-20T18:59:03.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-07-20T18:59:03.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]: dispatch 2023-07-20T18:59:03.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-07-20T18:59:03.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2023-07-20T18:59:03.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:02 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2023-07-20T18:59:03.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.88:6836/678288704,v1:172.21.15.88:6837/678288704] up:active 2023-07-20T18:59:03.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='client.? 172.21.15.57:0/3441015755' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T18:59:03.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T18:59:03.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-20T18:59:03.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-20T18:59:03.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-20T18:59:03.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-07-20T18:59:03.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2023-07-20T18:59:03.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2023-07-20T18:59:03.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-07-20T18:59:03.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2023-07-20T18:59:03.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-07-20T18:59:03.487 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]: dispatch 2023-07-20T18:59:03.488 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-07-20T18:59:03.488 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2023-07-20T18:59:03.488 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:02 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2023-07-20T18:59:03.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.523+0000 7f45e2dff700 1 -- 172.21.15.57:0/1509368740 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f45dc060c40 con 0x7f45cc0a11d0 2023-07-20T18:59:03.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.549+0000 7f45d9ffb700 1 -- 172.21.15.57:0/1509368740 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f45dc060c40 con 0x7f45cc0a11d0 2023-07-20T18:59:03.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.551+0000 7f45e2dff700 1 -- 172.21.15.57:0/1509368740 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45cc0a11d0 msgr2=0x7f45cc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:03.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.551+0000 7f45e2dff700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45cc0a11d0 0x7f45cc0a3690 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f45dc1017b0 tx=0x7f45d0009260 comp rx=0 tx=0).stop 2023-07-20T18:59:03.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.551+0000 7f45e2dff700 1 -- 172.21.15.57:0/1509368740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45dc101950 msgr2=0x7f45dc072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:03.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.551+0000 7f45e2dff700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45dc101950 0x7f45dc072b60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f45c800e9a0 tx=0x7f45c800ecb0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.551+0000 7f45e2dff700 1 -- 172.21.15.57:0/1509368740 shutdown_connections 2023-07-20T18:59:03.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.551+0000 7f45e2dff700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f45cc0a11d0 0x7f45cc0a3690 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.551+0000 7f45e2dff700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc100760 0x7f45dc074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.551+0000 7f45e2dff700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc1022f0 0x7f45dc0730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.551+0000 7f45e2dff700 1 --2- 172.21.15.57:0/1509368740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45dc101950 0x7f45dc072b60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.551+0000 7f45e2dff700 1 -- 172.21.15.57:0/1509368740 >> 172.21.15.57:0/1509368740 conn(0x7f45dc0fbcb0 msgr2=0x7f45dc0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:03.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.552+0000 7f45e2dff700 1 -- 172.21.15.57:0/1509368740 shutdown_connections 2023-07-20T18:59:03.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.552+0000 7f45e2dff700 1 -- 172.21.15.57:0/1509368740 wait complete. 2023-07-20T18:59:03.621 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2023-07-20T18:59:03.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.784+0000 7f893edff700 1 -- 172.21.15.57:0/4003292992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8938074750 msgr2=0x7f8938072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:03.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.784+0000 7f893edff700 1 --2- 172.21.15.57:0/4003292992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8938074750 0x7f8938072ba0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f892c00b0a0 tx=0x7f892c00b3b0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.784+0000 7f893edff700 1 -- 172.21.15.57:0/4003292992 shutdown_connections 2023-07-20T18:59:03.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.784+0000 7f893edff700 1 --2- 172.21.15.57:0/4003292992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8938108630 0x7f893810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.784+0000 7f893edff700 1 --2- 172.21.15.57:0/4003292992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8938074750 0x7f8938072ba0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.784+0000 7f893edff700 1 --2- 172.21.15.57:0/4003292992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8938073170 0x7f89380735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.784+0000 7f893edff700 1 -- 172.21.15.57:0/4003292992 >> 172.21.15.57:0/4003292992 conn(0x7f89380fbcb0 msgr2=0x7f89380fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:03.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.784+0000 7f893edff700 1 -- 172.21.15.57:0/4003292992 shutdown_connections 2023-07-20T18:59:03.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.784+0000 7f893edff700 1 -- 172.21.15.57:0/4003292992 wait complete. 2023-07-20T18:59:03.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.785+0000 7f893edff700 1 Processor -- start 2023-07-20T18:59:03.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.785+0000 7f893edff700 1 -- start start 2023-07-20T18:59:03.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.785+0000 7f893edff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8938073170 0x7f89381988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:03.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.785+0000 7f893edff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8938074750 0x7f8938198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:03.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.785+0000 7f893edff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8938108630 0x7f893819fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:03.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.785+0000 7f893edff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89381a0440 con 0x7f8938074750 2023-07-20T18:59:03.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.785+0000 7f893edff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89381a0580 con 0x7f8938108630 2023-07-20T18:59:03.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.785+0000 7f893edff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89381a0710 con 0x7f8938073170 2023-07-20T18:59:03.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f8937fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8938074750 0x7f8938198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:03.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f8937fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8938074750 0x7f8938198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40128/0 (socket says 172.21.15.57:40128) 2023-07-20T18:59:03.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f8937fff700 1 -- 172.21.15.57:0/1706095463 learned_addr learned my addr 172.21.15.57:0/1706095463 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-07-20T18:59:03.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f893cb9b700 1 --2- 172.21.15.57:0/1706095463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8938073170 0x7f89381988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:03.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f893d39c700 1 --2- 172.21.15.57:0/1706095463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8938108630 0x7f893819fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:03.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f8937fff700 1 -- 172.21.15.57:0/1706095463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8938073170 msgr2=0x7f89381988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:03.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f8937fff700 1 --2- 172.21.15.57:0/1706095463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8938073170 0x7f89381988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f8937fff700 1 -- 172.21.15.57:0/1706095463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8938108630 msgr2=0x7f893819fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:03.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f8937fff700 1 --2- 172.21.15.57:0/1706095463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8938108630 0x7f893819fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:03.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f8937fff700 1 -- 172.21.15.57:0/1706095463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c009d00 con 0x7f8938074750 2023-07-20T18:59:03.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f893d39c700 1 --2- 172.21.15.57:0/1706095463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8938108630 0x7f893819fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T18:59:03.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.786+0000 7f8937fff700 1 --2- 172.21.15.57:0/1706095463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8938074750 0x7f8938198e00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f892400e9a0 tx=0x7f892400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:03.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.787+0000 7f8935ffb700 1 -- 172.21.15.57:0/1706095463 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924009790 con 0x7f8938074750 2023-07-20T18:59:03.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.787+0000 7f8935ffb700 1 -- 172.21.15.57:0/1706095463 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(27 keys) v1 ==== 1139+0+0 (secure 0 0 0) 0x7f892400cbd0 con 0x7f8938074750 2023-07-20T18:59:03.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.787+0000 7f893edff700 1 -- 172.21.15.57:0/1706095463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89381a09f0 con 0x7f8938074750 2023-07-20T18:59:03.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.787+0000 7f8935ffb700 1 -- 172.21.15.57:0/1706095463 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924018680 con 0x7f8938074750 2023-07-20T18:59:03.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.787+0000 7f893edff700 1 -- 172.21.15.57:0/1706095463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89381a0f50 con 0x7f8938074750 2023-07-20T18:59:03.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.788+0000 7f893edff700 1 -- 172.21.15.57:0/1706095463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8938106d30 con 0x7f8938074750 2023-07-20T18:59:03.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.790+0000 7f8935ffb700 1 -- 172.21.15.57:0/1706095463 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f89240188a0 con 0x7f8938074750 2023-07-20T18:59:03.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.791+0000 7f8935ffb700 1 --2- 172.21.15.57:0/1706095463 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f89280a11d0 0x7f89280a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:03.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.791+0000 7f8935ffb700 1 -- 172.21.15.57:0/1706095463 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8924014070 con 0x7f8938074750 2023-07-20T18:59:03.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.791+0000 7f893cb9b700 1 --2- 172.21.15.57:0/1706095463 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f89280a11d0 0x7f89280a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:03.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.792+0000 7f893cb9b700 1 --2- 172.21.15.57:0/1706095463 >> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] conn(0x7f89280a11d0 0x7f89280a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f892c00b780 tx=0x7f892c006210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:03.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:03.795+0000 7f8935ffb700 1 -- 172.21.15.57:0/1706095463 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8924051780 con 0x7f8938074750 2023-07-20T18:59:04.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:04.049+0000 7f893edff700 1 -- 172.21.15.57:0/1706095463 --> [v2:172.21.15.57:6800/4059331298,v1:172.21.15.57:6801/4059331298] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f8938060c40 con 0x7f89280a11d0 2023-07-20T18:59:04.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-07-20T18:59:04.051+0000 7f8935ffb700 1 -- 172.21.15.57:0/1706095463 <== mgr.14150 v2:172.21.15.57:6800/4059331298 1 ==== mgr_command_reply(tid 0: -2 subvolume 'sv_0' does not exist) v1 ==== 39+0+0 (secure 0 0 0) 0x7f8938060c40 con 0x7f89280a11d0 2023-07-20T18:59:04.103 INFO:teuthology.orchestra.run.smithi057.stderr:Error ENOENT: subvolume 'sv_0' does not exist 2023-07-20T18:59:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-20T18:59:04.112 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' 2023-07-20T18:59:04.113 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T18:59:04.182 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T18:59:04.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T18:59:04.256 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-20T18:59:04.256 DEBUG:teuthology.orchestra.run.smithi057:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T18:59:04.299 DEBUG:teuthology.orchestra.run.smithi088:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T18:59:04.317 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T18:59:04.336 INFO:tasks.cephadm:Stopping all daemons... 2023-07-20T18:59:04.337 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-20T18:59:04.337 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.a 2023-07-20T18:59:04.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:59:04.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: pgmap v286: 129 pgs: 129 active+clean; 583 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2023-07-20T18:59:04.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-07-20T18:59:04.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2023-07-20T18:59:04.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2023-07-20T18:59:04.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-07-20T18:59:04.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2023-07-20T18:59:04.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-07-20T18:59:04.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]': finished 2023-07-20T18:59:04.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-07-20T18:59:04.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2023-07-20T18:59:04.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2023-07-20T18:59:04.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: osdmap e87: 12 total, 12 up, 12 in 2023-07-20T18:59:04.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.192:6836/2391717255,v1:172.21.15.192:6837/2391717255] up:active 2023-07-20T18:59:04.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T18:59:04.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T18:59:04.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: Cluster is now healthy 2023-07-20T18:59:04.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2023-07-20T18:59:04.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-20T18:59:04.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: daemon mds.e is now active in filesystem cephfs as rank 4 2023-07-20T18:59:04.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: mds.? [v2:172.21.15.88:6834/1373743783,v1:172.21.15.88:6835/1373743783] up:active 2023-07-20T18:59:04.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:03 smithi192 ceph-mon[129200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-20T18:59:04.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:59:04.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: pgmap v286: 129 pgs: 129 active+clean; 583 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2023-07-20T18:59:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-07-20T18:59:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2023-07-20T18:59:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2023-07-20T18:59:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-07-20T18:59:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2023-07-20T18:59:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-07-20T18:59:04.384 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]': finished 2023-07-20T18:59:04.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-07-20T18:59:04.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2023-07-20T18:59:04.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2023-07-20T18:59:04.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: osdmap e87: 12 total, 12 up, 12 in 2023-07-20T18:59:04.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.192:6836/2391717255,v1:172.21.15.192:6837/2391717255] up:active 2023-07-20T18:59:04.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T18:59:04.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T18:59:04.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: Cluster is now healthy 2023-07-20T18:59:04.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2023-07-20T18:59:04.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-20T18:59:04.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: daemon mds.e is now active in filesystem cephfs as rank 4 2023-07-20T18:59:04.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: mds.? [v2:172.21.15.88:6834/1373743783,v1:172.21.15.88:6835/1373743783] up:active 2023-07-20T18:59:04.386 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:03 smithi088 ceph-mon[128823]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-20T18:59:04.399 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:59:04 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:59:04.051+0000 7f038fe5e700 -1 mgr.server reply reply (2) No such file or directory subvolume 'sv_0' does not exist 2023-07-20T18:59:04.399 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T18:59:04.400 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: pgmap v286: 129 pgs: 129 active+clean; 583 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2023-07-20T18:59:04.400 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-07-20T18:59:04.400 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2023-07-20T18:59:04.400 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2023-07-20T18:59:04.400 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-07-20T18:59:04.400 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2023-07-20T18:59:04.401 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-07-20T18:59:04.401 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]': finished 2023-07-20T18:59:04.401 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-07-20T18:59:04.401 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2023-07-20T18:59:04.401 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: from='mgr.14150 172.21.15.57:0/1157359423' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2023-07-20T18:59:04.401 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: osdmap e87: 12 total, 12 up, 12 in 2023-07-20T18:59:04.401 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.192:6836/2391717255,v1:172.21.15.192:6837/2391717255] up:active 2023-07-20T18:59:04.401 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T18:59:04.402 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T18:59:04.402 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: Cluster is now healthy 2023-07-20T18:59:04.402 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2023-07-20T18:59:04.402 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-20T18:59:04.402 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: daemon mds.e is now active in filesystem cephfs as rank 4 2023-07-20T18:59:04.402 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: mds.? [v2:172.21.15.88:6834/1373743783,v1:172.21.15.88:6835/1373743783] up:active 2023-07-20T18:59:04.402 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:03 smithi057 ceph-mon[126340]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-20T18:59:04.736 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:04 smithi057 systemd[1]: Stopping Ceph mon.a for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:05.133 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:04 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-a[126316]: 2023-07-20T18:59:04.791+0000 7fc7ad79e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:05.133 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:04 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-a[126316]: 2023-07-20T18:59:04.791+0000 7fc7ad79e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-20T18:59:05.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Jul 20 18:59:05 smithi057 bash[164982]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-a 2023-07-20T18:59:05.673 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.a.service' 2023-07-20T18:59:05.804 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:05.805 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-20T18:59:05.805 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-20T18:59:05.805 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.b 2023-07-20T18:59:06.133 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:05 smithi088 systemd[1]: Stopping Ceph mon.b for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:06.133 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 18:59:05 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-b[154175]: 2023-07-20T18:59:05.773+0000 7fc19c7e5700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-07-20T18:59:06.134 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 18:59:05 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-b[154175]: 2023-07-20T18:59:05.773+0000 7fc19c7e5700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-07-20T18:59:06.134 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 18:59:05 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-b[154175]: 2023-07-20T18:59:05.773+0000 7fc19c7e5700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-07-20T18:59:06.630 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-b[128799]: 2023-07-20T18:59:06.158+0000 7f20b66c1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:06.630 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-b[128799]: 2023-07-20T18:59:06.158+0000 7f20b66c1700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-20T18:59:06.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 20 18:59:06 smithi088 bash[155429]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-b 2023-07-20T18:59:06.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.b.service' 2023-07-20T18:59:06.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:06.979 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-20T18:59:06.980 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-20T18:59:06.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.c 2023-07-20T18:59:07.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:06 smithi192 systemd[1]: Stopping Ceph mon.c for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:07.726 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:07 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-c[129176]: 2023-07-20T18:59:07.294+0000 7f4aad87e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:07.726 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:07 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-c[129176]: 2023-07-20T18:59:07.294+0000 7f4aad87e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-20T18:59:07.982 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jul 20 18:59:07 smithi192 bash[156850]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mon-c 2023-07-20T18:59:07.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mon.c.service' 2023-07-20T18:59:08.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:08.048 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-20T18:59:08.048 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-20T18:59:08.049 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T18:59:08.486 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:59:08 smithi057 systemd[1]: Stopping Ceph mgr.x for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:08.873 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:59:08 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:59:08.542+0000 7f0421a92700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:08.873 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:59:08 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x[126584]: 2023-07-20T18:59:08.542+0000 7f0421a92700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-07-20T18:59:09.236 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Jul 20 18:59:08 smithi057 bash[165489]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-x 2023-07-20T18:59:09.424 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.x.service' 2023-07-20T18:59:09.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:09.516 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-20T18:59:09.516 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-20T18:59:09.517 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.y 2023-07-20T18:59:09.884 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:59:09 smithi088 systemd[1]: Stopping Ceph mgr.y for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:10.227 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:59:10 smithi088 bash[155744]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-y 2023-07-20T18:59:10.228 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 20 18:59:10 smithi088 systemd[1]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-20T18:59:10.415 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.y.service' 2023-07-20T18:59:10.453 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:10.454 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-20T18:59:10.455 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-20T18:59:10.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.z 2023-07-20T18:59:10.840 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:59:10 smithi192 systemd[1]: Stopping Ceph mgr.z for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:11.242 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:59:10 smithi192 bash[157017]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mgr-z 2023-07-20T18:59:11.242 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Jul 20 18:59:11 smithi192 systemd[1]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-20T18:59:11.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mgr.z.service' 2023-07-20T18:59:11.285 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:11.288 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-20T18:59:11.288 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-20T18:59:11.289 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.0 2023-07-20T18:59:11.736 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:59:11 smithi057 systemd[1]: Stopping Ceph osd.0 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:12.236 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:59:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-0[139331]: 2023-07-20T18:59:11.764+0000 7fce2b638700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:12.236 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:59:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-0[139331]: 2023-07-20T18:59:11.764+0000 7fce2b638700 -1 osd.0 88 *** Got signal Terminated *** 2023-07-20T18:59:12.236 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:59:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-0[139331]: 2023-07-20T18:59:11.764+0000 7fce2b638700 -1 osd.0 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T18:59:17.986 INFO:journalctl@ceph.osd.0.smithi057.stdout:Jul 20 18:59:17 smithi057 bash[165653]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-0 2023-07-20T18:59:19.451 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.0.service' 2023-07-20T18:59:19.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:19.544 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-20T18:59:19.545 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-20T18:59:19.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.1 2023-07-20T18:59:19.884 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:59:19 smithi088 systemd[1]: Stopping Ceph osd.1 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:20.383 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:59:19 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-1[133946]: 2023-07-20T18:59:19.870+0000 7fa3945ea700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:20.384 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:59:19 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-1[133946]: 2023-07-20T18:59:19.870+0000 7fa3945ea700 -1 osd.1 88 *** Got signal Terminated *** 2023-07-20T18:59:20.384 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:59:19 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-1[133946]: 2023-07-20T18:59:19.870+0000 7fa3945ea700 -1 osd.1 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T18:59:25.883 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 20 18:59:25 smithi088 bash[155901]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-1 2023-07-20T18:59:26.948 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.1.service' 2023-07-20T18:59:26.988 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:26.989 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-20T18:59:26.989 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-20T18:59:26.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.2 2023-07-20T18:59:27.341 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:59:26 smithi192 systemd[1]: Stopping Ceph osd.2 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:27.841 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:59:27 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-2[135031]: 2023-07-20T18:59:27.299+0000 7f16649d2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:27.841 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:59:27 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-2[135031]: 2023-07-20T18:59:27.299+0000 7f16649d2700 -1 osd.2 88 *** Got signal Terminated *** 2023-07-20T18:59:27.841 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:59:27 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-2[135031]: 2023-07-20T18:59:27.299+0000 7f16649d2700 -1 osd.2 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T18:59:33.341 INFO:journalctl@ceph.osd.2.smithi192.stdout:Jul 20 18:59:32 smithi192 bash[157162]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-2 2023-07-20T18:59:34.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.2.service' 2023-07-20T18:59:34.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:34.475 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-20T18:59:34.476 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-20T18:59:34.476 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.3 2023-07-20T18:59:34.986 INFO:journalctl@ceph.osd.3.smithi057.stdout:Jul 20 18:59:34 smithi057 systemd[1]: Stopping Ceph osd.3 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:35.486 INFO:journalctl@ceph.osd.3.smithi057.stdout:Jul 20 18:59:34 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-3[144749]: 2023-07-20T18:59:34.954+0000 7fac23653700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:35.487 INFO:journalctl@ceph.osd.3.smithi057.stdout:Jul 20 18:59:34 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-3[144749]: 2023-07-20T18:59:34.954+0000 7fac23653700 -1 osd.3 88 *** Got signal Terminated *** 2023-07-20T18:59:35.487 INFO:journalctl@ceph.osd.3.smithi057.stdout:Jul 20 18:59:34 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-3[144749]: 2023-07-20T18:59:34.954+0000 7fac23653700 -1 osd.3 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T18:59:38.591 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:38 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:38.251+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:39.134 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:38 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:38.636+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:39.591 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:39 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:39.268+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:40.134 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:39 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:39.608+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:40.486 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:40 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:40.108+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:40.591 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:40.223+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:40.884 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:40 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:40.576+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:40.986 INFO:journalctl@ceph.osd.3.smithi057.stdout:Jul 20 18:59:40 smithi057 bash[165926]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-3 2023-07-20T18:59:40.987 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:40 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:40.724+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:41.091 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:40.687+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:41.486 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:41 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:41.128+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:41.591 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:41 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:41.263+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:41.987 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:41 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:41.774+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:42.091 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:41 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:41.654+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:42.091 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:41 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:41.791+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:42.134 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:41 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:41.598+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:42.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:42 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:42.082+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:42.591 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:42.256+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:42.704 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.3.service' 2023-07-20T18:59:42.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:42.796 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-20T18:59:42.796 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-20T18:59:42.796 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.4 2023-07-20T18:59:42.987 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:42 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:42.773+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:43.091 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:42.611+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:43.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:42.749+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:43.134 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 20 18:59:42 smithi088 systemd[1]: Stopping Ceph osd.4 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:43.134 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:42 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:42.610+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:43.486 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:43 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:43.044+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:43.628 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:43 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:43.299+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:43.884 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:43 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:43.577+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:43.884 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 20 18:59:43 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-4[139332]: 2023-07-20T18:59:43.339+0000 7f8ccde17700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:43.884 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 20 18:59:43 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-4[139332]: 2023-07-20T18:59:43.339+0000 7f8ccde17700 -1 osd.4 88 *** Got signal Terminated *** 2023-07-20T18:59:43.885 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 20 18:59:43 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-4[139332]: 2023-07-20T18:59:43.339+0000 7f8ccde17700 -1 osd.4 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T18:59:44.091 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:43 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:43.577+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:44.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:43 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:43.719+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:44.116 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:43 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:43.799+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:44.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:44 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:44.065+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:44.659 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:44 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:44.291+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:44.884 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:44 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:44.550+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:45.091 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:44 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:44.743+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:45.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:44 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:44.607+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:45.110 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:44 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:44.768+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:45.486 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:45 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:45.059+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:45.619 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:45 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:45.298+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:45.884 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:45 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:45.550+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:46.091 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:45 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:45.567+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:46.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:45 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:45.726+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:46.098 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:45 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:45.774+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:46.098 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:45 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:45.774+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:46.486 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:46 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:46.047+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:46.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:46 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:46.047+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:46.631 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:46 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:46.304+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:46.632 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:46 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:46.304+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:19.741884+0000 front 2023-07-20T18:59:19.741979+0000 (oldest deadline 2023-07-20T18:59:45.642003+0000) 2023-07-20T18:59:46.884 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:46 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:46.556+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:47.091 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:46 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:46.744+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:47.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:46 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:46.579+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:47.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:46 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:46.579+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:47.237 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:47 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:47.070+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:47.237 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:47 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:47.070+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:47.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:46 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:46.823+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:47.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:46 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:46.823+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:47.638 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:47 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:47.311+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:47.639 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:47 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:47.311+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:19.741884+0000 front 2023-07-20T18:59:19.741979+0000 (oldest deadline 2023-07-20T18:59:45.642003+0000) 2023-07-20T18:59:47.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:47 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:47.472+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:47.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:47 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:47.554+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:47.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:47 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:47.554+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:48.091 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:47 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:47.767+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:48.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:47 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:47.586+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:48.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:47 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:47.586+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:48.151 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:47 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:47.786+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:48.151 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:47 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:47.786+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:48.486 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:48 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:48.100+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:48.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:48 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:48.100+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:48.811 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:48 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:48.333+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:48.811 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:48 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:48.333+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:19.741884+0000 front 2023-07-20T18:59:19.741979+0000 (oldest deadline 2023-07-20T18:59:45.642003+0000) 2023-07-20T18:59:48.812 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:48 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:48.578+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:48.812 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:48 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:48.578+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:48.867 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:48 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:48.456+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:48.868 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:48 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:48.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:48.868 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:48 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:48.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:49.091 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:48 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:48.758+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:49.134 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 20 18:59:49 smithi088 bash[156169]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-4 2023-07-20T18:59:49.174 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:48 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:48.744+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:49.174 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:48 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:48.744+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:49.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:49 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:49.123+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:49.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:49 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:49.123+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:49.654 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:49 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:49.303+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:49.654 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:49 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:49.303+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:19.741884+0000 front 2023-07-20T18:59:19.741979+0000 (oldest deadline 2023-07-20T18:59:45.642003+0000) 2023-07-20T18:59:49.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:49 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:49.487+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:49.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:49 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:49.557+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:49.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:49 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:49.557+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:50.091 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:49 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:49.789+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:50.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:49 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:49.789+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T18:59:50.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:49 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:49.602+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:50.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:49 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:49.602+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:50.183 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:49 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:49.738+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:50.183 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:49 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:49.738+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:50.294 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.4.service' 2023-07-20T18:59:50.335 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:50.336 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-20T18:59:50.336 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-20T18:59:50.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.5 2023-07-20T18:59:50.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:50 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:50.132+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:50.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:50 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:50.132+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:50.591 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:50 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:50.285+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:50.592 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:50 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:50.285+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:19.741884+0000 front 2023-07-20T18:59:19.741979+0000 (oldest deadline 2023-07-20T18:59:45.642003+0000) 2023-07-20T18:59:50.592 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:50 smithi192 systemd[1]: Stopping Ceph osd.5 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:50.634 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:50 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:50.476+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:51.091 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:50 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:50.624+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:51.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:50 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:50.624+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:51.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:50 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:50.749+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:51.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:50 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:50.750+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T18:59:51.093 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:50 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:50.645+0000 7fb787812700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:51.094 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:50 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:50.645+0000 7fb787812700 -1 osd.5 88 *** Got signal Terminated *** 2023-07-20T18:59:51.094 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:50 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:50.645+0000 7fb787812700 -1 osd.5 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T18:59:51.134 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:50 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:50.597+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:51.134 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:50 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:50.597+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:51.232 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:50 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:50.754+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:51.232 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:50 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:50.754+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:51.486 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:51 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:51.181+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:51.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:51 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:51.181+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:51.591 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:51 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:51.259+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:51.592 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:51 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:51.259+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:19.741884+0000 front 2023-07-20T18:59:19.741979+0000 (oldest deadline 2023-07-20T18:59:45.642003+0000) 2023-07-20T18:59:51.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:51 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:51.466+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:51.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:51 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:51.587+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:51.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:51 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:51.587+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:52.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:51 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:51.657+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:52.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:51 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:51.657+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:52.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:51 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:51.753+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:52.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:51 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:51.753+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T18:59:52.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:51 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:51.793+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:52.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:51 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:51.793+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:52.591 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:52 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:52.271+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:52.592 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:52 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:52.271+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:19.741884+0000 front 2023-07-20T18:59:19.741979+0000 (oldest deadline 2023-07-20T18:59:45.642003+0000) 2023-07-20T18:59:52.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:52 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:52.217+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:52.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:52 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:52.217+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:52.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:52 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:52.461+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:52.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:52 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:52.623+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:52.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:52 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:52.623+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:53.091 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:52 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:52.706+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:53.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:52 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:52.706+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T18:59:53.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:52 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:52.670+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:53.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:52 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:52.670+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:53.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:52 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:52.803+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:53.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:52 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:52.803+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:53.698 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:53 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:53.320+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:53.698 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:53 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:53.320+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:19.741884+0000 front 2023-07-20T18:59:19.741979+0000 (oldest deadline 2023-07-20T18:59:45.642003+0000) 2023-07-20T18:59:53.744 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:53 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:53.221+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:53.744 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:53 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:53.221+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:53.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:53 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:53.508+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:53.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:53 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:53.575+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:53.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:53 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:53.575+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:54.091 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:53 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:53.720+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:54.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:53 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:53.720+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T18:59:54.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:53 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:53.646+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:54.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:53 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:53.646+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:54.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:53 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:53.647+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T18:59:54.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:53 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:53.808+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:54.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:53 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:53.808+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:54.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:54 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:54.188+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:54.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:54 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:54.188+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:54.746 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:54 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:54.312+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:54.746 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:54 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:54.313+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:19.741884+0000 front 2023-07-20T18:59:19.741979+0000 (oldest deadline 2023-07-20T18:59:45.642003+0000) 2023-07-20T18:59:54.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:54 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:54.460+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:54.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:54 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:54.543+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:54.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:54 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:54.543+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:54.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:54 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:54.543+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T18:59:55.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:54 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:54.695+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:55.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:54 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:54.695+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:55.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:54 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:54.695+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T18:59:55.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:54 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:54.766+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:55.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:54 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:54.766+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T18:59:55.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:54 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:54.766+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T18:59:55.231 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:54 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:54.778+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:55.231 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:54 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:54.778+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:55.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:55.180+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:55.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:55.180+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:55.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:55.180+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.903732+0000 front 2023-07-20T18:59:31.903924+0000 (oldest deadline 2023-07-20T18:59:54.203620+0000) 2023-07-20T18:59:55.591 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:55 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:55.284+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.040244+0000 front 2023-07-20T18:59:14.040196+0000 (oldest deadline 2023-07-20T18:59:38.140091+0000) 2023-07-20T18:59:55.592 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:55 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:55.284+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:19.741884+0000 front 2023-07-20T18:59:19.741979+0000 (oldest deadline 2023-07-20T18:59:45.642003+0000) 2023-07-20T18:59:55.592 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:55 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5[140567]: 2023-07-20T18:59:55.284+0000 7fb781e05700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.843540+0000 front 2023-07-20T18:59:30.843482+0000 (oldest deadline 2023-07-20T18:59:54.343750+0000) 2023-07-20T18:59:55.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:55 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:55.479+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:55.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:55 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:55.479+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T18:59:55.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:55 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:55.594+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:55.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:55 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:55.594+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:55.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:55 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:55.594+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T18:59:56.088 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:55 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:55.701+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:56.088 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:55 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:55.701+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:56.089 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:55 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:55.701+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T18:59:56.089 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:55 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:55.729+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:56.089 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:55 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:55.729+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T18:59:56.089 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:55 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:55.729+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T18:59:56.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:55.772+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:56.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:55.772+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:56.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:55 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:55.772+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T18:59:56.341 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 20 18:59:56 smithi192 bash[157440]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-5 2023-07-20T18:59:56.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:56 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:56.212+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:56.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:56 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:56.212+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:56.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:56 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:56.212+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.903732+0000 front 2023-07-20T18:59:31.903924+0000 (oldest deadline 2023-07-20T18:59:54.203620+0000) 2023-07-20T18:59:56.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:56 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:56.516+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:56.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:56 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:56.516+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T18:59:56.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:56 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:56.549+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:56.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:56 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:56.549+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:56.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:56 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:56.549+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T18:59:57.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:56 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:56.707+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:57.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:56 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:56.707+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:57.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:56 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:56.707+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T18:59:57.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:56 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:56.726+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:57.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:56 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:56.726+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T18:59:57.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:56 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:56.726+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T18:59:57.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:56 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:56.732+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:57.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:56 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:56.732+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:57.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:56 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:56.732+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T18:59:57.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:57 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:57.230+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:57.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:57 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:57.230+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:57.738 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:57 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:57.230+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.903732+0000 front 2023-07-20T18:59:31.903924+0000 (oldest deadline 2023-07-20T18:59:54.203620+0000) 2023-07-20T18:59:57.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.5.service' 2023-07-20T18:59:57.782 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T18:59:57.782 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-20T18:59:57.783 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-20T18:59:57.783 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.6 2023-07-20T18:59:57.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:57 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:57.552+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:57.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:57 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:57.552+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T18:59:57.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:57 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:57.590+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:57.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:57 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:57.590+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:57.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:57 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:57.590+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T18:59:58.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:57 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:57.750+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:58.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:57 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:57.750+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:58.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:57 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:57.750+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T18:59:58.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:57 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:57.748+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:58.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:57 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:57.748+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T18:59:58.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:57 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:57.748+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T18:59:58.237 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:57 smithi057 systemd[1]: Stopping Ceph osd.6 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T18:59:58.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:57 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:57.730+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:58.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:57 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:57.730+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:58.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:57 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:57.730+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T18:59:58.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:58.258+0000 7ff59f6cf700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:58.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:58.258+0000 7ff59f6cf700 -1 osd.6 88 *** Got signal Terminated *** 2023-07-20T18:59:58.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:58.258+0000 7ff59f6cf700 -1 osd.6 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T18:59:58.738 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:58.266+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:58.738 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:58.266+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:58.738 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:58.266+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.903732+0000 front 2023-07-20T18:59:31.903924+0000 (oldest deadline 2023-07-20T18:59:54.203620+0000) 2023-07-20T18:59:58.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:58 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:58.596+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:58.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:58 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:58.596+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T18:59:58.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:58 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:58.577+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:58.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:58 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:58.577+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:58.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:58 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:58.577+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T18:59:59.091 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:58 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:58.796+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T18:59:59.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:58 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:58.796+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T18:59:59.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:58 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:58.796+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T18:59:59.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:58 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:58.756+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T18:59:59.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:58 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:58.756+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T18:59:59.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:58 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:58.756+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T18:59:59.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:58.689+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T18:59:59.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:58.689+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T18:59:59.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:58 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:58.689+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T18:59:59.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:59 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:59.303+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T18:59:59.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:59 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:59.303+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T18:59:59.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 18:59:59 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T18:59:59.303+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.903732+0000 front 2023-07-20T18:59:31.903924+0000 (oldest deadline 2023-07-20T18:59:54.203620+0000) 2023-07-20T18:59:59.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:59 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:59.571+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T18:59:59.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 18:59:59 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T18:59:59.571+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T18:59:59.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:59 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:59.547+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T18:59:59.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:59 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:59.547+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T18:59:59.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 18:59:59 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T18:59:59.547+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:00.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:59 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:59.723+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:00.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:59 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:59.723+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:00.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 18:59:59 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T18:59:59.723+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:00.342 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:59 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:59.800+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:00.342 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:59 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:59.800+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:00.342 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 18:59:59 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T18:59:59.800+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:00.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:59 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:59.796+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:00.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:59 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:59.796+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:00.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 18:59:59 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T18:59:59.796+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:00.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 19:00:00 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T19:00:00.288+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T19:00:00.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 19:00:00 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T19:00:00.288+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T19:00:00.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 19:00:00 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T19:00:00.288+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.903732+0000 front 2023-07-20T18:59:31.903924+0000 (oldest deadline 2023-07-20T18:59:54.203620+0000) 2023-07-20T19:00:00.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:00 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:00.605+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:00.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:00 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:00.605+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:00.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:00 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:00.577+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:00.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:00 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:00.577+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:00.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:00 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:00.577+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:01.091 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:00 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:00.824+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:01.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:00 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:00.824+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:01.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:00 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:00.824+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:01.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:00 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:00.748+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:01.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:00 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:00.748+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:01.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:00 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:00.748+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:01.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:00 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:00.755+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:01.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:00 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:00.755+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:01.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:00 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:00.755+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:01.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 19:00:01 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T19:00:01.315+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T19:00:01.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 19:00:01 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T19:00:01.315+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T19:00:01.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 19:00:01 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T19:00:01.315+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.903732+0000 front 2023-07-20T18:59:31.903924+0000 (oldest deadline 2023-07-20T18:59:54.203620+0000) 2023-07-20T19:00:01.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:01 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:01.602+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:01.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:01 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:01.602+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:01.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:01 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:01.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:01.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:01 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:01.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:01.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:01 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:01.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:01.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:01 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:01.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:02.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:01 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:01.800+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:02.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:01 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:01.800+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:02.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:01 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:01.800+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:02.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:01 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:01.800+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:02.342 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:01 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:01.870+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:02.342 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:01 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:01.870+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:02.342 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:01 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:01.870+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:02.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:01 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:01.791+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:02.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:01 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:01.791+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:02.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:01 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:01.791+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:02.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 19:00:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T19:00:02.304+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.299090+0000 front 2023-07-20T18:59:16.298862+0000 (oldest deadline 2023-07-20T18:59:39.699843+0000) 2023-07-20T19:00:02.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 19:00:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T19:00:02.304+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:22.601392+0000 front 2023-07-20T18:59:22.601150+0000 (oldest deadline 2023-07-20T18:59:45.501376+0000) 2023-07-20T19:00:02.737 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 19:00:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6[150222]: 2023-07-20T19:00:02.304+0000 7ff59bcc6700 -1 osd.6 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.903732+0000 front 2023-07-20T18:59:31.903924+0000 (oldest deadline 2023-07-20T18:59:54.203620+0000) 2023-07-20T19:00:02.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:02 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:02.570+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:02.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:02 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:02.570+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:02.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:02 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:02.504+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:02.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:02 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:02.504+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:02.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:02 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:02.504+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:02.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:02 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:02.504+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:03.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:02.822+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:03.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:02.822+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:03.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:02.822+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:03.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:02 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:02.822+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:03.342 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:02 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:02.874+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:03.342 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:02 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:02.874+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:03.342 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:02 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:02.874+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:03.342 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:02 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:02.874+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:03.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:02 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:02.806+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:03.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:02 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:02.806+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:03.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:02 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:02.806+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:03.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:02 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:02.806+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:03.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:03 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:03.567+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:03.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:03 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:03.567+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:03.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:03 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:03.567+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.608456+0000 front 2023-07-20T18:59:39.608268+0000 (oldest deadline 2023-07-20T19:00:03.108157+0000) 2023-07-20T19:00:03.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:03 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:03.532+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:03.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:03 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:03.532+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:03.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:03 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:03.532+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:03.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:03 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:03.532+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:03.987 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:03 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:03.774+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:03.987 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:03 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:03.774+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:03.987 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:03 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:03.774+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:03.987 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:03 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:03.774+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:04.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:03 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:03.908+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:04.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:03 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:03.908+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:04.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:03 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:03.908+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:04.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:03 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:03.908+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:04.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:03 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:03.773+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:04.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:03 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:03.773+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:04.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:03 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:03.773+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:04.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:03 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:03.773+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:04.487 INFO:journalctl@ceph.osd.6.smithi057.stdout:Jul 20 19:00:03 smithi057 bash[166204]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-6 2023-07-20T19:00:04.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:04 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:04.555+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:04.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:04 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:04.555+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:04.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:04 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:04.555+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.608456+0000 front 2023-07-20T18:59:39.608268+0000 (oldest deadline 2023-07-20T19:00:03.108157+0000) 2023-07-20T19:00:04.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:04 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:04.496+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:04.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:04 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:04.496+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:04.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:04 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:04.496+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:04.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:04 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:04.496+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:05.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:04 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:04.916+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:05.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:04 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:04.916+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:05.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:04 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:04.916+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:05.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:04 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:04.916+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:05.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:04 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:04.776+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:05.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:04 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:04.776+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:05.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:04 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:04.776+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:05.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:04 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:04.776+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:05.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:04 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:04.813+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:05.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:04 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:04.813+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:05.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:04 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:04.813+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:05.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:04 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:04.813+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:05.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:05 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:05.580+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:05.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:05 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:05.580+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:05.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:05 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:05.580+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.608456+0000 front 2023-07-20T18:59:39.608268+0000 (oldest deadline 2023-07-20T19:00:03.108157+0000) 2023-07-20T19:00:05.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:05 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:05.544+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:05.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:05 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:05.544+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:05.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:05 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:05.544+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:05.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:05 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:05.544+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:05.928 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.6.service' 2023-07-20T19:00:06.018 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:00:06.019 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-20T19:00:06.019 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-20T19:00:06.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.7 2023-07-20T19:00:06.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:05 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:05.908+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:06.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:05 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:05.908+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:06.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:05 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:05.908+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:06.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:05 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:05.908+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:06.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:05 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:05.785+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:06.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:05 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:05.785+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:06.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:05 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:05.785+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:06.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:05 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:05.785+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:06.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:05 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:05.801+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:06.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:05 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:05.801+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:06.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:05 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:05.801+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:06.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:05 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:05.801+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:06.384 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:06 smithi088 systemd[1]: Stopping Ceph osd.7 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:00:06.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:06.566+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:06.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:06.566+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:06.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:06.566+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:06.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:06.566+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:06.886 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:06.334+0000 7f92007be700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:00:06.886 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:06.334+0000 7f92007be700 -1 osd.7 88 *** Got signal Terminated *** 2023-07-20T19:00:06.886 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:06.334+0000 7f92007be700 -1 osd.7 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:00:06.887 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:06.558+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:06.887 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:06.558+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:06.887 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:06 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:06.558+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.608456+0000 front 2023-07-20T18:59:39.608268+0000 (oldest deadline 2023-07-20T19:00:03.108157+0000) 2023-07-20T19:00:07.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:06 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:06.864+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:07.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:06 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:06.864+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:07.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:06 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:06.864+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:07.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:06 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:06.864+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:07.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:06 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:06.774+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:07.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:06 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:06.774+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:07.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:06 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:06.774+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:07.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:06 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:06.774+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:07.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:06 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:06.828+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:07.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:06 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:06.828+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:07.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:06 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:06.828+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:07.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:06 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:06.828+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:07.884 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:07 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:07.608+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:07.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:07 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:07.608+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:07.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:07 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:07.608+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:07.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:07 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:07.608+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:07.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:07 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:07.557+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:07.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:07 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:07.557+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:07.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:07 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:07.557+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.608456+0000 front 2023-07-20T18:59:39.608268+0000 (oldest deadline 2023-07-20T19:00:03.108157+0000) 2023-07-20T19:00:07.936 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:07 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:07.877+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:08.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:07 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:07.862+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:08.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:07 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:07.862+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:08.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:07 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:07.862+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:08.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:07 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:07.862+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:08.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:07 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:07.785+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:08.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:07 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:07.785+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:08.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:07 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:07.785+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:08.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:07 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:07.785+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:08.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:07 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:07.877+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:08.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:07 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:07.877+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:08.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:07 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:07.877+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:08.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:08 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:08.541+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:08.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:08 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:08.541+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:08.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:08 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:08.541+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.608456+0000 front 2023-07-20T18:59:39.608268+0000 (oldest deadline 2023-07-20T19:00:03.108157+0000) 2023-07-20T19:00:08.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:08 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:08.652+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:08.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:08 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:08.652+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:08.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:08 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:08.652+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:08.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:08 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:08.652+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:09.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:08 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:08.860+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:09.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:08 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:08.860+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:09.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:08 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:08.860+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:09.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:08 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:08.860+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:09.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:08 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:08.746+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:09.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:08 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:08.746+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:09.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:08 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:08.746+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:09.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:08 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:08.746+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:09.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:08 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:08.859+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:09.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:08 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:08.859+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:09.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:08 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:08.859+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:09.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:08 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:08.859+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:09.884 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:09 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:09.553+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:09.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:09 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:09.553+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:09.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:09 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:09.553+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.608456+0000 front 2023-07-20T18:59:39.608268+0000 (oldest deadline 2023-07-20T19:00:03.108157+0000) 2023-07-20T19:00:09.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:09 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:09.617+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:09.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:09 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:09.617+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:09.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:09 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:09.617+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:09.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:09 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:09.617+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:09.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:09 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:09.617+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:10.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:09 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:09.821+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:10.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:09 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:09.821+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:10.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:09 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:09.821+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:10.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:09 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:09.821+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:10.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:09 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:09.790+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:10.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:09 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:09.790+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:10.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:09 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:09.790+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:10.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:09 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:09.790+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:10.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:09 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:09.856+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:10.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:09 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:09.856+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:10.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:09 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:09.856+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:10.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:09 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:09.856+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:10.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:09 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:09.856+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:10.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:10 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:10.504+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.803238+0000 front 2023-07-20T18:59:24.803217+0000 (oldest deadline 2023-07-20T18:59:46.503382+0000) 2023-07-20T19:00:10.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:10 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:10.504+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.807244+0000 front 2023-07-20T18:59:30.807014+0000 (oldest deadline 2023-07-20T18:59:54.907038+0000) 2023-07-20T19:00:10.885 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:10 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7[144845]: 2023-07-20T19:00:10.504+0000 7f91fa5b0700 -1 osd.7 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.608456+0000 front 2023-07-20T18:59:39.608268+0000 (oldest deadline 2023-07-20T19:00:03.108157+0000) 2023-07-20T19:00:10.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:10 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:10.607+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:10.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:10 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:10.607+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:10.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:10 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:10.607+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:10.887 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:10 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:10.607+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:10.887 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:10 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:10.607+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:11.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:10.772+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:11.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:10.772+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:11.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:10.772+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:11.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:10.772+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:11.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:10.772+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:45.131022+0000 front 2023-07-20T18:59:45.131115+0000 (oldest deadline 2023-07-20T19:00:10.331544+0000) 2023-07-20T19:00:11.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:10.835+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:11.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:10.835+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:11.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:10.835+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:11.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:10.835+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:11.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:10 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:10.835+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:11.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:10 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:10.875+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:11.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:10 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:10.875+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:11.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:10 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:10.875+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:11.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:10 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:10.875+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:11.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:10 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:10.875+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:12.042 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:11 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:11.644+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:12.043 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:11 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:11.644+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:12.043 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:11 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:11.644+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:12.043 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:11 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:11.644+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:12.043 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:11 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:11.644+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:12.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:11.727+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:12.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:11.727+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:12.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:11.727+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:12.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:11.727+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:12.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:11.727+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:45.131022+0000 front 2023-07-20T18:59:45.131115+0000 (oldest deadline 2023-07-20T19:00:10.331544+0000) 2023-07-20T19:00:12.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:11.798+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:12.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:11.798+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:12.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:11.798+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:12.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:11.798+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:12.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:11 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:11.798+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:12.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:11.904+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:12.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:11.904+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:12.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:11.904+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:12.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:11.904+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:12.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:11 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:11.904+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:12.384 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 20 19:00:11 smithi088 bash[156432]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-7 2023-07-20T19:00:12.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:12.639+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:12.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:12.639+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:12.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:12.639+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:12.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:12.639+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:12.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:12.639+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:13.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:12.736+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:13.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:12.736+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:13.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:12.736+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:13.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:12.736+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:13.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:12.736+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:45.131022+0000 front 2023-07-20T18:59:45.131115+0000 (oldest deadline 2023-07-20T19:00:10.331544+0000) 2023-07-20T19:00:13.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:12.787+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:13.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:12.787+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:13.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:12.787+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:13.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:12.787+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:13.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:12 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:12.787+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:13.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:12 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:12.919+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:13.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:12 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:12.919+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:13.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:12 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:12.919+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:13.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:12 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:12.919+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:13.239 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:12 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:12.919+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:13.424 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.7.service' 2023-07-20T19:00:13.476 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:00:13.477 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-20T19:00:13.477 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-20T19:00:13.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.8 2023-07-20T19:00:13.792 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:13 smithi192 systemd[1]: Stopping Ceph osd.8 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:00:14.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:13.740+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:14.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:13.740+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:14.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:13.740+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:14.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:13.740+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:14.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:13.740+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:45.131022+0000 front 2023-07-20T18:59:45.131115+0000 (oldest deadline 2023-07-20T19:00:10.331544+0000) 2023-07-20T19:00:14.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:13.801+0000 7f1d729e4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:00:14.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:13.801+0000 7f1d729e4700 -1 osd.8 88 *** Got signal Terminated *** 2023-07-20T19:00:14.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:13.801+0000 7f1d729e4700 -1 osd.8 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:00:14.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:13.810+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:14.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:13.810+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:14.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:13.810+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:14.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:13.810+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:14.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:13 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:13.810+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:14.134 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:13 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:13.669+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:14.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:13 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:13.669+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:14.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:13 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:13.669+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:14.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:13 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:13.669+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:14.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:13 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:13.669+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:14.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:13 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:13.876+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:14.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:13 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:13.876+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:14.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:13 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:13.876+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:14.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:13 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:13.876+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:14.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:13 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:13.876+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:15.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:14.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:15.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:14.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:15.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:14.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:15.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:14.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:15.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:14.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:15.094 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:14.700+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:15.094 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:14.700+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:15.095 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:14.700+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:15.095 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:14.700+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:15.095 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:14 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:14.700+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:45.131022+0000 front 2023-07-20T18:59:45.131115+0000 (oldest deadline 2023-07-20T19:00:10.331544+0000) 2023-07-20T19:00:15.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:14 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:14.674+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:15.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:14 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:14.674+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:15.136 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:14 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:14.674+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:15.136 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:14 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:14.674+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:15.136 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:14 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:14.674+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:15.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:14 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:14.909+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:15.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:14 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:14.909+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:15.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:14 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:14.909+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:15.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:14 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:14.909+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:15.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:14 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:14.909+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:16.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:15.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:16.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:15.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:16.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:15.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:16.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:15.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:16.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:15.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:16.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:15.668+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:16.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:15.668+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:16.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:15.668+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:16.094 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:15.668+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:16.094 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:15 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:15.668+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:45.131022+0000 front 2023-07-20T18:59:45.131115+0000 (oldest deadline 2023-07-20T19:00:10.331544+0000) 2023-07-20T19:00:16.134 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:15.644+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:16.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:15.644+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:16.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:15.644+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:16.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:15.644+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:16.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:15 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:15.644+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:16.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:15 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:15.885+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:16.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:15 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:15.885+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:16.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:15 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:15.885+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:16.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:15 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:15.885+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:16.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:15 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:15.885+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:17.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:16.664+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:17.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:16.664+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:17.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:16.664+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:17.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:16.664+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:17.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:16.664+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:45.131022+0000 front 2023-07-20T18:59:45.131115+0000 (oldest deadline 2023-07-20T19:00:10.331544+0000) 2023-07-20T19:00:17.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:16.807+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:17.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:16.807+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:17.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:16.807+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:17.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:16.807+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:17.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:16 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:16.807+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:17.134 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:16 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:16.638+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:17.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:16 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:16.638+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:17.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:16 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:16.638+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:17.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:16 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:16.638+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:17.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:16 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:16.638+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:17.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:16 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:16.928+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:17.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:16 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:16.928+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:17.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:16 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:16.928+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:17.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:16 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:16.928+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:17.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:16 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:16.928+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:18.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:17.633+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:18.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:17.633+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:18.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:17.633+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:18.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:17.633+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:18.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:17.633+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:45.131022+0000 front 2023-07-20T18:59:45.131115+0000 (oldest deadline 2023-07-20T19:00:10.331544+0000) 2023-07-20T19:00:18.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:17.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:18.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:17.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:18.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:17.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:18.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:17.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:18.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:17.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:18.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:17 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:17.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:18.134 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:17 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:17.632+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:18.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:17 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:17.632+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:18.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:17 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:17.632+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:18.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:17 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:17.632+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:18.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:17 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:17.632+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:18.487 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:17 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:17.967+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:18.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:17 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:17.967+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:18.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:17 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:17.967+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:18.489 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:17 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:17.967+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:18.489 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:17 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:17.967+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:19.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:18.628+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:14.825976+0000 front 2023-07-20T18:59:14.826011+0000 (oldest deadline 2023-07-20T18:59:40.125877+0000) 2023-07-20T19:00:19.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:18.628+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.126415+0000 front 2023-07-20T18:59:20.126552+0000 (oldest deadline 2023-07-20T18:59:46.026416+0000) 2023-07-20T19:00:19.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:18.628+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:31.227853+0000 front 2023-07-20T18:59:31.227877+0000 (oldest deadline 2023-07-20T18:59:53.528064+0000) 2023-07-20T19:00:19.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:18.628+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:39.929797+0000 front 2023-07-20T18:59:39.929722+0000 (oldest deadline 2023-07-20T19:00:02.229872+0000) 2023-07-20T19:00:19.093 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8[146078]: 2023-07-20T19:00:18.628+0000 7f1d6cfd7700 -1 osd.8 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:45.131022+0000 front 2023-07-20T18:59:45.131115+0000 (oldest deadline 2023-07-20T19:00:10.331544+0000) 2023-07-20T19:00:19.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:18.773+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:19.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:18.773+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:19.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:18.773+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:19.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:18.773+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:19.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:18.773+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:19.094 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:18 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:18.773+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:19.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:18 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:18.585+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:19.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:18 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:18.585+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:19.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:18 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:18.585+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:19.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:18 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:18.585+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:19.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:18 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:18.585+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:19.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:18 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:18.933+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:19.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:18 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:18.933+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:19.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:18 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:18.933+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:19.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:18 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:18.933+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:19.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:18 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:18.933+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:19.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:18 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:18.933+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.913737+0000 front 2023-07-20T18:59:54.913767+0000 (oldest deadline 2023-07-20T19:00:18.413944+0000) 2023-07-20T19:00:19.842 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jul 20 19:00:19 smithi192 bash[157704]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-8 2023-07-20T19:00:19.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:19 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:19.560+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:19.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:19 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:19.560+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:19.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:19 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:19.560+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:19.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:19 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:19.560+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:19.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:19 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:19.560+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:20.181 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:19 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:19.805+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:20.182 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:19 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:19.805+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:20.182 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:19 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:19.805+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:20.182 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:19 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:19.805+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:20.182 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:19 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:19.805+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:20.182 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:19 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:19.805+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:20.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:19 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:19.920+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:20.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:19 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:19.920+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:20.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:19 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:19.920+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:20.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:19 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:19.920+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:20.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:19 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:19.920+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:20.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:19 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:19.920+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.913737+0000 front 2023-07-20T18:59:54.913767+0000 (oldest deadline 2023-07-20T19:00:18.413944+0000) 2023-07-20T19:00:20.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:20 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:20.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:20.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:20 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:20.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:20.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:20 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:20.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:20.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:20 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:20.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:20.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:20 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:20.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:20.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:20 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:20.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:20.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.8.service' 2023-07-20T19:00:21.007 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:00:21.008 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-20T19:00:21.008 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-20T19:00:21.009 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.9 2023-07-20T19:00:21.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:20 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:20.781+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:21.092 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:20 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:20.781+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:21.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:20 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:20.781+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:21.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:20 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:20.781+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:21.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:20 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:20.782+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:21.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:20 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:20.782+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:21.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:20 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:20.950+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:21.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:20 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:20.950+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:21.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:20 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:20.950+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:21.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:20 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:20.950+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:21.489 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:20 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:20.950+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:21.489 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:20 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:20.950+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.913737+0000 front 2023-07-20T18:59:54.913767+0000 (oldest deadline 2023-07-20T19:00:18.413944+0000) 2023-07-20T19:00:21.489 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:21 smithi057 systemd[1]: Stopping Ceph osd.9 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:00:21.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:21 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:21.521+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:21.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:21 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:21.521+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:21.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:21 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:21.521+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:21.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:21 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:21.521+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:21.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:21 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:21.521+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:21.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:21 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:21.521+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:21.987 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:21 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:21.508+0000 7efd72402700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:00:21.988 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:21 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:21.508+0000 7efd72402700 -1 osd.9 88 *** Got signal Terminated *** 2023-07-20T19:00:21.988 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:21 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:21.508+0000 7efd72402700 -1 osd.9 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:00:22.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:21 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:21.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:22.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:21 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:21.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:22.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:21 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:21.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:22.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:21 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:21.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:22.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:21 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:21.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:22.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:21 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:21.800+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:22.487 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:21 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:21.943+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:22.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:21 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:21.943+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:22.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:21 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:21.943+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:22.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:21 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:21.943+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:22.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:21 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:21.943+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:22.489 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:21 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:21.943+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.913737+0000 front 2023-07-20T18:59:54.913767+0000 (oldest deadline 2023-07-20T19:00:18.413944+0000) 2023-07-20T19:00:22.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:22 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:22.518+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:22.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:22 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:22.518+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:22.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:22 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:22.518+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:22.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:22 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:22.518+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:22.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:22 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:22.518+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:22.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:22 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:22.518+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:23.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:22 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:22.900+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:23.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:22 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:22.900+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:23.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:22 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:22.900+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:23.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:22 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:22.900+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:23.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:22 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:22.900+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:23.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:22 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:22.900+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.913737+0000 front 2023-07-20T18:59:54.913767+0000 (oldest deadline 2023-07-20T19:00:18.413944+0000) 2023-07-20T19:00:23.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:22 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:22.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:23.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:22 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:22.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:23.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:22 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:22.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:23.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:22 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:22.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:23.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:22 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:22.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:23.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:22 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:22.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:23.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:23 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:23.532+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:23.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:23 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:23.532+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:23.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:23 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:23.532+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:23.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:23 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:23.532+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:23.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:23 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:23.532+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:23.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:23 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:23.532+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:24.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:23 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:23.874+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:24.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:23 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:23.874+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:24.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:23 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:23.874+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:24.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:23 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:23.874+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:24.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:23 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:23.874+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:24.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:23 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:23.874+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.913737+0000 front 2023-07-20T18:59:54.913767+0000 (oldest deadline 2023-07-20T19:00:18.413944+0000) 2023-07-20T19:00:24.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:23 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:23.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:24.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:23 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:23.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:24.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:23 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:23.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:24.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:23 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:23.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:24.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:23 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:23.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:24.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:23 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:23.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:24.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:24 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:24.503+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:24.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:24 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:24.503+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:24.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:24 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:24.503+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:24.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:24 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:24.503+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:24.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:24 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:24.503+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:24.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:24 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:24.503+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:25.237 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:24 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:24.917+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:25.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:24 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:24.917+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:25.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:24 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:24.917+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:25.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:24 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:24.917+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:25.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:24 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:24.917+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:25.238 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:24 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:24.917+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.913737+0000 front 2023-07-20T18:59:54.913767+0000 (oldest deadline 2023-07-20T19:00:18.413944+0000) 2023-07-20T19:00:25.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:24 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:24.828+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:25.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:24 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:24.828+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:25.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:24 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:24.828+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:25.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:24 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:24.828+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:25.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:24 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:24.828+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:25.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:24 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:24.828+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:25.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:25 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:25.508+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:25.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:25 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:25.508+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:25.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:25 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:25.508+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:25.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:25 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:25.508+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:25.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:25 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:25.508+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:25.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:25 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:25.508+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:25.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:25 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:25.508+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:59.749579+0000 front 2023-07-20T18:59:59.749732+0000 (oldest deadline 2023-07-20T19:00:25.049329+0000) 2023-07-20T19:00:26.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:25 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:25.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:26.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:25 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:25.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:26.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:25 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:25.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:26.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:25 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:25.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:26.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:25 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:25.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:26.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:25 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:25.824+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:26.487 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:25 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:25.947+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.405477+0000 front 2023-07-20T18:59:16.405737+0000 (oldest deadline 2023-07-20T18:59:40.505739+0000) 2023-07-20T19:00:26.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:25 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:25.947+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:24.006863+0000 front 2023-07-20T18:59:24.006936+0000 (oldest deadline 2023-07-20T18:59:45.707156+0000) 2023-07-20T19:00:26.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:25 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:25.947+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:30.309186+0000 front 2023-07-20T18:59:30.309454+0000 (oldest deadline 2023-07-20T18:59:55.609333+0000) 2023-07-20T19:00:26.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:25 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:25.947+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:35.609914+0000 front 2023-07-20T18:59:35.609947+0000 (oldest deadline 2023-07-20T19:00:01.510183+0000) 2023-07-20T19:00:26.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:25 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:25.947+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:46.112810+0000 front 2023-07-20T18:59:46.112682+0000 (oldest deadline 2023-07-20T19:00:09.012663+0000) 2023-07-20T19:00:26.488 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:25 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9[155759]: 2023-07-20T19:00:25.947+0000 7efd6b9f3700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.913737+0000 front 2023-07-20T18:59:54.913767+0000 (oldest deadline 2023-07-20T19:00:18.413944+0000) 2023-07-20T19:00:26.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:26 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:26.550+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:26.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:26 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:26.550+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:26.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:26 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:26.550+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:26.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:26 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:26.550+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:26.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:26 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:26.550+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:26.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:26 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:26.550+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:26.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:26 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:26.550+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:59.749579+0000 front 2023-07-20T18:59:59.749732+0000 (oldest deadline 2023-07-20T19:00:25.049329+0000) 2023-07-20T19:00:27.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:26 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:26.851+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:27.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:26 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:26.851+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:27.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:26 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:26.851+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:27.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:26 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:26.851+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:27.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:26 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:26.851+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:27.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:26 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:26.851+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:27.416 INFO:journalctl@ceph.osd.9.smithi057.stdout:Jul 20 19:00:27 smithi057 bash[166474]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-9 2023-07-20T19:00:27.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:27 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:27.561+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:27.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:27 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:27.561+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:27.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:27 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:27.561+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:27.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:27 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:27.561+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:27.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:27 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:27.561+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:27.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:27 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:27.561+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:27.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:27 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:27.561+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:59.749579+0000 front 2023-07-20T18:59:59.749732+0000 (oldest deadline 2023-07-20T19:00:25.049329+0000) 2023-07-20T19:00:28.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:27 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:27.817+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:28.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:27 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:27.817+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:28.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:27 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:27.817+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:28.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:27 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:27.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:28.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:27 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:27.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:28.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:27 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:27.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:28.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:27 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:27.818+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:28.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:28 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:28.570+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:28.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:28 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:28.570+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:28.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:28 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:28.570+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:28.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:28 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:28.570+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:28.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:28 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:28.570+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:28.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:28 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:28.570+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:28.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:28 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:28.570+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:59.749579+0000 front 2023-07-20T18:59:59.749732+0000 (oldest deadline 2023-07-20T19:00:25.049329+0000) 2023-07-20T19:00:29.253 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.9.service' 2023-07-20T19:00:29.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:28 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:28.852+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:29.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:28 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:28.852+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:29.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:28 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:28.852+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:29.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:28 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:28.852+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:29.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:28 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:28.852+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:29.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:28 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:28.852+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:29.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:28 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:28.852+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:29.361 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:00:29.362 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-20T19:00:29.362 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-20T19:00:29.363 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.10 2023-07-20T19:00:29.722 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 systemd[1]: Stopping Ceph osd.10 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:00:29.723 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:29.527+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:29.723 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:29.527+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:29.723 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:29.527+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:29.723 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:29.527+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:29.723 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:29.527+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:29.724 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:29.527+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:29.724 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:29.527+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:59.749579+0000 front 2023-07-20T18:59:59.749732+0000 (oldest deadline 2023-07-20T19:00:25.049329+0000) 2023-07-20T19:00:30.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:29.670+0000 7fa163b28700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:00:30.135 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:29.670+0000 7fa163b28700 -1 osd.10 88 *** Got signal Terminated *** 2023-07-20T19:00:30.136 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:29 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:29.670+0000 7fa163b28700 -1 osd.10 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:00:30.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:29 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:29.902+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:30.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:29 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:29.902+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:30.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:29 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:29.902+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:30.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:29 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:29.902+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:30.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:29 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:29.902+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:30.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:29 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:29.902+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:30.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:29 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:29.902+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:30.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:30 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:30.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:30.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:30 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:30.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:30.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:30 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:30.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:30.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:30 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:30.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:30.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:30 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:30.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:30.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:30 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:30.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:30.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:30 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:30.542+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:59.749579+0000 front 2023-07-20T18:59:59.749732+0000 (oldest deadline 2023-07-20T19:00:25.049329+0000) 2023-07-20T19:00:31.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:30 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:30.925+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:31.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:30 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:30.925+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:31.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:30 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:30.925+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:31.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:30 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:30.925+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:31.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:30 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:30.925+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:31.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:30 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:30.925+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:31.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:30 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:30.925+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:31.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:31 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:31.537+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:31.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:31 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:31.537+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:31.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:31 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:31.537+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:31.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:31 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:31.537+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:31.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:31 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:31.537+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:31.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:31 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:31.537+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:31.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:31 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:31.537+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:59.749579+0000 front 2023-07-20T18:59:59.749732+0000 (oldest deadline 2023-07-20T19:00:25.049329+0000) 2023-07-20T19:00:32.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:31 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:31.900+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:32.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:31 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:31.900+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:32.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:31 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:31.900+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:32.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:31 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:31.900+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:32.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:31 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:31.900+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:32.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:31 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:31.900+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:32.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:31 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:31.900+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:32.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:32 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:32.520+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:32.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:32 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:32.520+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:32.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:32 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:32.520+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:32.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:32 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:32.520+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:32.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:32 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:32.520+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:32.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:32 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:32.520+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:32.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:32 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:32.520+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:59.749579+0000 front 2023-07-20T18:59:59.749732+0000 (oldest deadline 2023-07-20T19:00:25.049329+0000) 2023-07-20T19:00:33.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:32 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:32.871+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:33.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:32 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:32.871+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:33.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:32 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:32.872+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:33.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:32 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:32.872+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:33.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:32 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:32.872+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:33.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:32 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:32.872+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:33.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:32 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:32.872+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:33.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:32 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:32.872+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-20T19:00:07.079247+0000 front 2023-07-20T19:00:07.079310+0000 (oldest deadline 2023-07-20T19:00:32.379266+0000) 2023-07-20T19:00:33.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:33 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:33.557+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:33.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:33 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:33.558+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:33.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:33 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:33.558+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:33.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:33 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:33.558+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:33.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:33 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:33.558+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:33.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:33 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:33.558+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:33.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:33 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:33.558+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:59.749579+0000 front 2023-07-20T18:59:59.749732+0000 (oldest deadline 2023-07-20T19:00:25.049329+0000) 2023-07-20T19:00:34.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:33 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:33.863+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:34.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:33 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:33.863+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:34.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:33 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:33.863+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:34.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:33 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:33.863+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:34.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:33 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:33.863+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:34.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:33 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:33.863+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:34.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:33 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:33.863+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:34.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:33 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:33.863+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-20T19:00:07.079247+0000 front 2023-07-20T19:00:07.079310+0000 (oldest deadline 2023-07-20T19:00:32.379266+0000) 2023-07-20T19:00:34.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:34 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:34.543+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:12.740425+0000 front 2023-07-20T18:59:12.740441+0000 (oldest deadline 2023-07-20T18:59:38.040421+0000) 2023-07-20T19:00:34.885 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:34 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:34.543+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:23.742573+0000 front 2023-07-20T18:59:23.742621+0000 (oldest deadline 2023-07-20T18:59:46.642853+0000) 2023-07-20T19:00:34.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:34 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:34.543+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:28.944144+0000 front 2023-07-20T18:59:28.944027+0000 (oldest deadline 2023-07-20T18:59:53.643939+0000) 2023-07-20T19:00:34.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:34 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:34.543+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:37.745146+0000 front 2023-07-20T18:59:37.745077+0000 (oldest deadline 2023-07-20T19:00:01.245080+0000) 2023-07-20T19:00:34.886 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:34 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:34.543+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:47.547275+0000 front 2023-07-20T18:59:47.547302+0000 (oldest deadline 2023-07-20T19:00:09.247406+0000) 2023-07-20T19:00:34.887 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:34 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:34.543+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:55.148520+0000 front 2023-07-20T18:59:55.148667+0000 (oldest deadline 2023-07-20T19:00:19.748943+0000) 2023-07-20T19:00:34.887 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:34 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10[150392]: 2023-07-20T19:00:34.543+0000 7fa15e11b700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:59.749579+0000 front 2023-07-20T18:59:59.749732+0000 (oldest deadline 2023-07-20T19:00:25.049329+0000) 2023-07-20T19:00:35.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:34 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:34.886+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:35.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:34 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:34.886+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:35.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:34 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:34.886+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:35.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:34 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:34.886+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:35.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:34 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:34.886+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:35.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:34 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:34.886+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:35.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:34 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:34.886+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:35.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:34 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:34.886+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-20T19:00:07.079247+0000 front 2023-07-20T19:00:07.079310+0000 (oldest deadline 2023-07-20T19:00:32.379266+0000) 2023-07-20T19:00:35.635 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 20 19:00:35 smithi088 bash[156705]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-10 2023-07-20T19:00:36.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:35 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:35.871+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:36.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:35 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:35.871+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:36.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:35 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:35.871+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:36.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:35 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:35.871+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:36.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:35 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:35.871+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:36.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:35 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:35.871+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:36.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:35 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:35.871+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:36.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:35 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:35.871+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-20T19:00:07.079247+0000 front 2023-07-20T19:00:07.079310+0000 (oldest deadline 2023-07-20T19:00:32.379266+0000) 2023-07-20T19:00:36.810 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.10.service' 2023-07-20T19:00:36.848 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:00:36.849 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-20T19:00:36.850 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-20T19:00:36.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.11 2023-07-20T19:00:37.207 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:36 smithi192 systemd[1]: Stopping Ceph osd.11 for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:00:37.207 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:36 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:36.907+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:37.208 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:36 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:36.907+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:37.208 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:36 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:36.907+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:37.208 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:36 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:36.907+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:37.208 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:36 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:36.907+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:37.208 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:36 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:36.907+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:37.208 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:36 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:36.907+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:37.208 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:36 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:36.907+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-20T19:00:07.079247+0000 front 2023-07-20T19:00:07.079310+0000 (oldest deadline 2023-07-20T19:00:32.379266+0000) 2023-07-20T19:00:37.209 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.154+0000 7f74c2c44700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:00:37.209 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.154+0000 7f74c2c44700 -1 osd.11 88 *** Got signal Terminated *** 2023-07-20T19:00:37.592 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.154+0000 7f74c2c44700 -1 osd.11 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:00:38.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.942+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:38.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.942+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:38.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.942+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:38.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.942+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:38.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.942+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:38.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.942+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:38.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.942+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:38.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:37.942+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-20T19:00:07.079247+0000 front 2023-07-20T19:00:07.079310+0000 (oldest deadline 2023-07-20T19:00:32.379266+0000) 2023-07-20T19:00:39.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:38 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:38.963+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:39.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:38 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:38.963+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:39.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:38 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:38.963+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:39.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:38 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:38.963+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:39.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:38 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:38.963+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:39.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:38 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:38.963+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:39.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:38 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:38.963+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:39.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:38 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:38.963+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-20T19:00:07.079247+0000 front 2023-07-20T19:00:07.079310+0000 (oldest deadline 2023-07-20T19:00:32.379266+0000) 2023-07-20T19:00:40.342 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:39 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:39.940+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:40.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:39 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:39.940+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:40.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:39 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:39.940+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:40.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:39 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:39.940+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:40.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:39 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:39.940+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:40.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:39 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:39.940+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:40.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:39 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:39.940+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:40.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:39 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:39.940+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-20T19:00:07.079247+0000 front 2023-07-20T19:00:07.079310+0000 (oldest deadline 2023-07-20T19:00:32.379266+0000) 2023-07-20T19:00:41.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:40.987+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:41.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:40.987+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:41.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:40.987+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:41.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:40.987+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:41.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:40.987+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:41.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:40.987+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:41.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:40.987+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:41.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:40.987+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-20T19:00:07.079247+0000 front 2023-07-20T19:00:07.079310+0000 (oldest deadline 2023-07-20T19:00:32.379266+0000) 2023-07-20T19:00:41.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:40 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:40.987+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6820 osd.8 since back 2023-07-20T19:00:18.180868+0000 front 2023-07-20T19:00:18.180905+0000 (oldest deadline 2023-07-20T19:00:40.480929+0000) 2023-07-20T19:00:42.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:42.006+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2023-07-20T18:59:16.271755+0000 front 2023-07-20T18:59:16.271665+0000 (oldest deadline 2023-07-20T18:59:40.971419+0000) 2023-07-20T19:00:42.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:42.006+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-20T18:59:20.972085+0000 front 2023-07-20T18:59:20.971948+0000 (oldest deadline 2023-07-20T18:59:49.172654+0000) 2023-07-20T19:00:42.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:42.006+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2023-07-20T18:59:29.173072+0000 front 2023-07-20T18:59:29.173105+0000 (oldest deadline 2023-07-20T18:59:53.873220+0000) 2023-07-20T19:00:42.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:42.006+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2023-07-20T18:59:36.174398+0000 front 2023-07-20T18:59:36.174554+0000 (oldest deadline 2023-07-20T19:00:02.074379+0000) 2023-07-20T19:00:42.343 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:42.006+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-20T18:59:43.175386+0000 front 2023-07-20T18:59:43.175333+0000 (oldest deadline 2023-07-20T19:00:10.176156+0000) 2023-07-20T19:00:42.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:42.006+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2023-07-20T18:59:54.777723+0000 front 2023-07-20T18:59:54.777658+0000 (oldest deadline 2023-07-20T19:00:17.677744+0000) 2023-07-20T19:00:42.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:42.006+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2023-07-20T18:59:57.678303+0000 front 2023-07-20T18:59:57.678314+0000 (oldest deadline 2023-07-20T19:00:27.078824+0000) 2023-07-20T19:00:42.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:42.006+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-20T19:00:07.079247+0000 front 2023-07-20T19:00:07.079310+0000 (oldest deadline 2023-07-20T19:00:32.379266+0000) 2023-07-20T19:00:42.344 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:42 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11[151496]: 2023-07-20T19:00:42.006+0000 7f74bea3a700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.192:6820 osd.8 since back 2023-07-20T19:00:18.180868+0000 front 2023-07-20T19:00:18.180905+0000 (oldest deadline 2023-07-20T19:00:40.480929+0000) 2023-07-20T19:00:43.093 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jul 20 19:00:42 smithi192 bash[157972]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-osd-11 2023-07-20T19:00:44.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@osd.11.service' 2023-07-20T19:00:44.414 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:00:44.414 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-20T19:00:44.415 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-20T19:00:44.415 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.a 2023-07-20T19:00:44.738 INFO:journalctl@ceph.mds.a.smithi057.stdout:Jul 20 19:00:44 smithi057 systemd[1]: Stopping Ceph mds.a for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:00:45.238 INFO:journalctl@ceph.mds.a.smithi057.stdout:Jul 20 19:00:44 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-a[161174]: 2023-07-20T19:00:44.884+0000 7f635d20c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:00:45.238 INFO:journalctl@ceph.mds.a.smithi057.stdout:Jul 20 19:00:44 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-a[161174]: 2023-07-20T19:00:44.884+0000 7f635d20c700 -1 mds.a *** got signal Terminated *** 2023-07-20T19:00:47.946 INFO:journalctl@ceph.mds.a.smithi057.stdout:Jul 20 19:00:47 smithi057 bash[166739]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-a 2023-07-20T19:00:47.971 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.a.service' 2023-07-20T19:00:48.056 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:00:48.057 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-20T19:00:48.057 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-20T19:00:48.058 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.d 2023-07-20T19:00:48.238 INFO:journalctl@ceph.mds.d.smithi057.stdout:Jul 20 19:00:48 smithi057 systemd[1]: Stopping Ceph mds.d for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:00:48.988 INFO:journalctl@ceph.mds.d.smithi057.stdout:Jul 20 19:00:48 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-d[160216]: 2023-07-20T19:00:48.508+0000 7efce3dc2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:00:48.988 INFO:journalctl@ceph.mds.d.smithi057.stdout:Jul 20 19:00:48 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-d[160216]: 2023-07-20T19:00:48.508+0000 7efce3dc2700 -1 mds.d *** got signal Terminated *** 2023-07-20T19:00:50.738 INFO:journalctl@ceph.mds.d.smithi057.stdout:Jul 20 19:00:50 smithi057 bash[166883]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-d 2023-07-20T19:00:50.906 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.d.service' 2023-07-20T19:00:50.990 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:00:50.991 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-20T19:00:50.991 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-20T19:00:50.992 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.g 2023-07-20T19:00:51.238 INFO:journalctl@ceph.mds.g.smithi057.stdout:Jul 20 19:00:51 smithi057 systemd[1]: Stopping Ceph mds.g for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:00:51.988 INFO:journalctl@ceph.mds.g.smithi057.stdout:Jul 20 19:00:51 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-g[160786]: 2023-07-20T19:00:51.450+0000 7fb4465a4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:00:51.988 INFO:journalctl@ceph.mds.g.smithi057.stdout:Jul 20 19:00:51 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-g[160786]: 2023-07-20T19:00:51.450+0000 7fb4465a4700 -1 mds.g *** got signal Terminated *** 2023-07-20T19:00:56.238 INFO:journalctl@ceph.mds.g.smithi057.stdout:Jul 20 19:00:55 smithi057 bash[167026]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-g 2023-07-20T19:00:56.280 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.g.service' 2023-07-20T19:00:56.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:00:56.364 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-20T19:00:56.364 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-20T19:00:56.365 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.j 2023-07-20T19:00:56.738 INFO:journalctl@ceph.mds.j.smithi057.stdout:Jul 20 19:00:56 smithi057 systemd[1]: Stopping Ceph mds.j for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:00:57.238 INFO:journalctl@ceph.mds.j.smithi057.stdout:Jul 20 19:00:56 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-j[159822]: 2023-07-20T19:00:56.816+0000 7f6a7dd5d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:00:57.239 INFO:journalctl@ceph.mds.j.smithi057.stdout:Jul 20 19:00:56 smithi057 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-j[159822]: 2023-07-20T19:00:56.816+0000 7f6a7dd5d700 -1 mds.j *** got signal Terminated *** 2023-07-20T19:00:59.738 INFO:journalctl@ceph.mds.j.smithi057.stdout:Jul 20 19:00:59 smithi057 bash[167168]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-j 2023-07-20T19:01:00.007 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.j.service' 2023-07-20T19:01:00.092 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:01:00.093 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-20T19:01:00.093 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-20T19:01:00.093 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.b 2023-07-20T19:01:00.385 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 19:01:00 smithi088 systemd[1]: Stopping Ceph mds.b for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:01:00.885 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 19:01:00 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-b[154175]: 2023-07-20T19:01:00.398+0000 7fc1a07ed700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:01:00.886 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 19:01:00 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-b[154175]: 2023-07-20T19:01:00.398+0000 7fc1a07ed700 -1 mds.b *** got signal Terminated *** 2023-07-20T19:01:07.519 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 20 19:01:07 smithi088 bash[156977]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-b 2023-07-20T19:01:07.524 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.b.service' 2023-07-20T19:01:07.562 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:01:07.563 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-20T19:01:07.563 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-20T19:01:07.563 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.e 2023-07-20T19:01:07.885 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 20 19:01:07 smithi088 systemd[1]: Stopping Ceph mds.e for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:01:08.386 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 20 19:01:07 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-e[153483]: 2023-07-20T19:01:07.887+0000 7f364778c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:01:08.386 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 20 19:01:07 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-e[153483]: 2023-07-20T19:01:07.887+0000 7f364778c700 -1 mds.e *** got signal Terminated *** 2023-07-20T19:01:12.316 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 20 19:01:12 smithi088 bash[157130]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-e 2023-07-20T19:01:12.509 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.e.service' 2023-07-20T19:01:12.543 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:01:12.544 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-20T19:01:12.544 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-20T19:01:12.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.h 2023-07-20T19:01:12.886 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 20 19:01:12 smithi088 systemd[1]: Stopping Ceph mds.h for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:01:13.386 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 20 19:01:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-h[153830]: 2023-07-20T19:01:12.868+0000 7f01e6083700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:01:13.386 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 20 19:01:12 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-h[153830]: 2023-07-20T19:01:12.868+0000 7f01e6083700 -1 mds.h *** got signal Terminated *** 2023-07-20T19:01:18.636 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 20 19:01:18 smithi088 bash[157267]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-h 2023-07-20T19:01:18.772 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.h.service' 2023-07-20T19:01:18.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:01:18.809 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-20T19:01:18.810 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-20T19:01:18.810 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.k 2023-07-20T19:01:19.136 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 20 19:01:18 smithi088 systemd[1]: Stopping Ceph mds.k for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:01:19.636 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 20 19:01:19 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-k[153141]: 2023-07-20T19:01:19.134+0000 7f5686870700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:01:19.636 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 20 19:01:19 smithi088 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-k[153141]: 2023-07-20T19:01:19.134+0000 7f5686870700 -1 mds.k *** got signal Terminated *** 2023-07-20T19:01:20.763 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 20 19:01:20 smithi088 bash[157421]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-k 2023-07-20T19:01:20.899 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.k.service' 2023-07-20T19:01:20.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:01:20.934 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-20T19:01:20.934 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-20T19:01:20.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.c 2023-07-20T19:01:21.294 INFO:journalctl@ceph.mds.c.smithi192.stdout:Jul 20 19:01:20 smithi192 systemd[1]: Stopping Ceph mds.c for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:01:21.593 INFO:journalctl@ceph.mds.c.smithi192.stdout:Jul 20 19:01:21 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-c[155088]: 2023-07-20T19:01:21.239+0000 7fdd76580700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:01:21.594 INFO:journalctl@ceph.mds.c.smithi192.stdout:Jul 20 19:01:21 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-c[155088]: 2023-07-20T19:01:21.239+0000 7fdd76580700 -1 mds.c *** got signal Terminated *** 2023-07-20T19:01:27.593 INFO:journalctl@ceph.mds.c.smithi192.stdout:Jul 20 19:01:27 smithi192 bash[158261]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-c 2023-07-20T19:01:27.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.c.service' 2023-07-20T19:01:27.677 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:01:27.678 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-20T19:01:27.678 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-20T19:01:27.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.f 2023-07-20T19:01:28.060 INFO:journalctl@ceph.mds.f.smithi192.stdout:Jul 20 19:01:27 smithi192 systemd[1]: Stopping Ceph mds.f for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:01:28.343 INFO:journalctl@ceph.mds.f.smithi192.stdout:Jul 20 19:01:28 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-f[154748]: 2023-07-20T19:01:28.005+0000 7f1e7d47b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:01:28.344 INFO:journalctl@ceph.mds.f.smithi192.stdout:Jul 20 19:01:28 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-f[154748]: 2023-07-20T19:01:28.005+0000 7f1e7d47b700 -1 mds.f *** got signal Terminated *** 2023-07-20T19:01:30.922 INFO:journalctl@ceph.mds.f.smithi192.stdout:Jul 20 19:01:30 smithi192 bash[158408]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-f 2023-07-20T19:01:31.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.f.service' 2023-07-20T19:01:31.146 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:01:31.146 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-20T19:01:31.147 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-20T19:01:31.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.i 2023-07-20T19:01:31.524 INFO:journalctl@ceph.mds.i.smithi192.stdout:Jul 20 19:01:31 smithi192 systemd[1]: Stopping Ceph mds.i for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:01:31.843 INFO:journalctl@ceph.mds.i.smithi192.stdout:Jul 20 19:01:31 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-i[155428]: 2023-07-20T19:01:31.470+0000 7f1801c62700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:01:31.844 INFO:journalctl@ceph.mds.i.smithi192.stdout:Jul 20 19:01:31 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-i[155428]: 2023-07-20T19:01:31.470+0000 7f1801c62700 -1 mds.i *** got signal Terminated *** 2023-07-20T19:01:37.409 INFO:journalctl@ceph.mds.i.smithi192.stdout:Jul 20 19:01:37 smithi192 bash[158548]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-i 2023-07-20T19:01:37.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.i.service' 2023-07-20T19:01:37.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:01:37.630 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-20T19:01:37.630 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-20T19:01:37.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.l 2023-07-20T19:01:37.844 INFO:journalctl@ceph.mds.l.smithi192.stdout:Jul 20 19:01:37 smithi192 systemd[1]: Stopping Ceph mds.l for 38a58c3c-272e-11ee-9b34-001a4aab830c... 2023-07-20T19:01:38.343 INFO:journalctl@ceph.mds.l.smithi192.stdout:Jul 20 19:01:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-l[154411]: 2023-07-20T19:01:37.963+0000 7f1ad8f1f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:01:38.344 INFO:journalctl@ceph.mds.l.smithi192.stdout:Jul 20 19:01:37 smithi192 ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-l[154411]: 2023-07-20T19:01:37.963+0000 7f1ad8f1f700 -1 mds.l *** got signal Terminated *** 2023-07-20T19:01:42.484 INFO:journalctl@ceph.mds.l.smithi192.stdout:Jul 20 19:01:42 smithi192 bash[158689]: ceph-38a58c3c-272e-11ee-9b34-001a4aab830c-mds-l 2023-07-20T19:01:42.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38a58c3c-272e-11ee-9b34-001a4aab830c@mds.l.service' 2023-07-20T19:01:42.716 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:01:42.717 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-20T19:01:42.717 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T19:01:43.119 INFO:teuthology.orchestra.run.smithi057.stdout:Deleting cluster with fsid: 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T19:01:46.472 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T19:01:46.773 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T19:01:48.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T19:01:48.628 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T19:01:50.197 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:50.278 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:50.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:50.344 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-20T19:01:50.346 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/remote/smithi057/crash 2023-07-20T19:01:50.347 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T19:01:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T19:01:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now 2023-07-20T19:01:50.428 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/remote/smithi088/crash 2023-07-20T19:01:50.429 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T19:01:50.464 INFO:teuthology.orchestra.run.smithi088.stderr:tar: /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T19:01:50.465 INFO:teuthology.orchestra.run.smithi088.stderr:tar: Error is not recoverable: exiting now 2023-07-20T19:01:50.466 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/remote/smithi192/crash 2023-07-20T19:01:50.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T19:01:50.507 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T19:01:50.507 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2023-07-20T19:01:50.508 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-20T19:01:50.508 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-20T19:01:50.593 INFO:teuthology.orchestra.run.smithi057.stderr:grep: /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph.log: No such file or directory 2023-07-20T19:01:50.598 INFO:tasks.cephadm:Compressing logs... 2023-07-20T19:01:50.598 DEBUG:teuthology.orchestra.run.smithi057:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T19:01:50.603 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T19:01:50.605 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T19:01:50.632 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T19:01:50.632 INFO:teuthology.orchestra.run.smithi192.stderr:: No such file or directory 2023-07-20T19:01:50.632 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.5.log 2023-07-20T19:01:50.633 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.2.log 2023-07-20T19:01:50.633 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.11.log 2023-07-20T19:01:50.633 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mgr.z.log 2023-07-20T19:01:50.634 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.8.log 2023-07-20T19:01:50.634 INFO:teuthology.orchestra.run.smithi192.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T19:01:50.634 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.f.log 2023-07-20T19:01:50.635 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.c.log 2023-07-20T19:01:50.635 INFO:teuthology.orchestra.run.smithi088.stderr:find: ‘/var/log/rbd-target-api’gzip -5 : No such file or directory--verbose 2023-07-20T19:01:50.635 INFO:teuthology.orchestra.run.smithi088.stderr:-- /var/log/ceph/cephadm.log 2023-07-20T19:01:50.636 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.10.log 2023-07-20T19:01:50.636 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.h.log 2023-07-20T19:01:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.e.log 2023-07-20T19:01:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.4.log 2023-07-20T19:01:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.e.log: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T19:01:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.1.log 2023-07-20T19:01:50.638 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.f.log: /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.c.log: 91.4% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mgr.z.log.gz 2023-07-20T19:01:50.638 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.i.log 2023-07-20T19:01:50.639 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.k.log 2023-07-20T19:01:50.639 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T19:01:50.640 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.i.log: 93.8% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.f.log.gz 2023-07-20T19:01:50.641 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mon.c.log 2023-07-20T19:01:50.642 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mon.c.log: 93.4% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.c.log.gz 2023-07-20T19:01:50.643 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.k.log: /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-volume.log: 93.5% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.e.log.gz 2023-07-20T19:01:50.643 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T19:01:50.644 INFO:teuthology.orchestra.run.smithi088.stderr: 94.1% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.k.log.gz 2023-07-20T19:01:50.645 INFO:teuthology.orchestra.run.smithi088.stderr: 93.1% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.h.log.gz 2023-07-20T19:01:50.646 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-volume.log: 93.7% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.i.log.gz 2023-07-20T19:01:50.647 INFO:teuthology.orchestra.run.smithi088.stderr: 91.5% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T19:01:50.647 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.7.log 2023-07-20T19:01:50.647 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mon.b.log 2023-07-20T19:01:50.648 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.b.log 2023-07-20T19:01:50.648 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mgr.y.log 2023-07-20T19:01:50.650 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.l.log 2023-07-20T19:01:50.651 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.l.log: 91.4% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T19:01:50.652 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.b.log: /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mgr.y.log: 91.3% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mgr.y.log.gz 2023-07-20T19:01:50.656 INFO:teuthology.orchestra.run.smithi192.stderr: 94.1% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.l.log.gz 2023-07-20T19:01:50.667 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- : No such file or directory/var/log/ceph/cephadm.log 2023-07-20T19:01:50.667 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-07-20T19:01:50.668 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.0.log 2023-07-20T19:01:50.669 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.g.log 2023-07-20T19:01:50.670 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.a.log 2023-07-20T19:01:50.672 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.6.log 2023-07-20T19:01:50.673 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.3.log 2023-07-20T19:01:50.674 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.6.log: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T19:01:50.675 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.d.log 2023-07-20T19:01:50.676 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.3.log: 94.0% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.g.log.gz 2023-07-20T19:01:50.676 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mgr.x.log 2023-07-20T19:01:50.677 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.9.log 2023-07-20T19:01:50.678 INFO:teuthology.orchestra.run.smithi057.stderr: 93.8% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.a.log.gz 2023-07-20T19:01:50.679 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mgr.x.log: gzip -5 --verbose -- 94.7%/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T19:01:50.679 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.d.log.gz 2023-07-20T19:01:50.680 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.j.log 2023-07-20T19:01:50.682 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mon.a.log 2023-07-20T19:01:50.684 INFO:teuthology.orchestra.run.smithi088.stderr: 93.8% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.b.log.gz 2023-07-20T19:01:50.685 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.j.log: /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2023-07-20T19:01:50.686 INFO:teuthology.orchestra.run.smithi057.stderr: 94.9% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mds.j.log.gz 2023-07-20T19:01:50.689 INFO:teuthology.orchestra.run.smithi057.stderr: 91.5% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T19:01:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.log: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2023-07-20T19:01:50.765 INFO:teuthology.orchestra.run.smithi088.stderr: 91.9% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mon.b.log.gz 2023-07-20T19:01:50.768 INFO:teuthology.orchestra.run.smithi192.stderr: 92.5% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.11.log.gz 2023-07-20T19:01:50.775 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mgr.x.log.gz 2023-07-20T19:01:50.803 INFO:teuthology.orchestra.run.smithi088.stderr: 92.1% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.10.log.gz 2023-07-20T19:01:50.818 INFO:teuthology.orchestra.run.smithi192.stderr: 91.9% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mon.c.log.gz 2023-07-20T19:01:50.839 INFO:teuthology.orchestra.run.smithi192.stderr: 92.0% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.5.log.gz 2023-07-20T19:01:50.846 INFO:teuthology.orchestra.run.smithi088.stderr: 92.1% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.7.log.gz 2023-07-20T19:01:50.860 INFO:teuthology.orchestra.run.smithi088.stderr: 92.2% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.4.log.gz 2023-07-20T19:01:50.861 INFO:teuthology.orchestra.run.smithi192.stderr: 91.9% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.2.log.gz 2023-07-20T19:01:50.862 INFO:teuthology.orchestra.run.smithi192.stderr: 92.2% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.8.log.gz 2023-07-20T19:01:50.863 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-07-20T19:01:50.863 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m0.242s 2023-07-20T19:01:50.863 INFO:teuthology.orchestra.run.smithi192.stderr:user 0m0.991s 2023-07-20T19:01:50.863 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m0.051s 2023-07-20T19:01:50.900 INFO:teuthology.orchestra.run.smithi057.stderr: 92.3% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.9.log.gz 2023-07-20T19:01:50.901 INFO:teuthology.orchestra.run.smithi057.stderr: 92.1% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.6.log.gz 2023-07-20T19:01:50.927 INFO:teuthology.orchestra.run.smithi088.stderr: 91.8% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.1.log.gz 2023-07-20T19:01:50.929 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-07-20T19:01:50.929 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m0.307s 2023-07-20T19:01:50.929 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m1.034s 2023-07-20T19:01:50.930 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m0.061s 2023-07-20T19:01:50.962 INFO:teuthology.orchestra.run.smithi057.stderr: 92.0% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.3.log.gz 2023-07-20T19:01:50.970 INFO:teuthology.orchestra.run.smithi057.stderr: 92.1% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-osd.0.log.gz 2023-07-20T19:01:51.036 INFO:teuthology.orchestra.run.smithi057.stderr: 91.0% -- replaced with /var/log/ceph/38a58c3c-272e-11ee-9b34-001a4aab830c/ceph-mon.a.log.gz 2023-07-20T19:01:51.039 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-07-20T19:01:51.039 INFO:teuthology.orchestra.run.smithi057.stderr:real 0m0.400s 2023-07-20T19:01:51.039 INFO:teuthology.orchestra.run.smithi057.stderr:user 0m1.400s 2023-07-20T19:01:51.039 INFO:teuthology.orchestra.run.smithi057.stderr:sys 0m0.199s 2023-07-20T19:01:51.040 INFO:tasks.cephadm:Archiving logs... 2023-07-20T19:01:51.041 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/remote/smithi057/log 2023-07-20T19:01:51.042 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T19:01:51.272 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/remote/smithi088/log 2023-07-20T19:01:51.273 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T19:01:51.407 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/remote/smithi192/log 2023-07-20T19:01:51.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T19:01:51.544 INFO:tasks.cephadm:Removing cluster... 2023-07-20T19:01:51.545 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c --force 2023-07-20T19:01:51.933 INFO:teuthology.orchestra.run.smithi057.stdout:Deleting cluster with fsid: 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T19:01:52.649 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c --force 2023-07-20T19:01:52.950 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T19:01:53.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38a58c3c-272e-11ee-9b34-001a4aab830c --force 2023-07-20T19:01:53.700 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: 38a58c3c-272e-11ee-9b34-001a4aab830c 2023-07-20T19:01:54.164 INFO:tasks.cephadm:Removing cephadm ... 2023-07-20T19:01:54.164 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T19:01:54.214 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T19:01:54.234 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T19:01:54.252 INFO:tasks.cephadm:Teardown complete 2023-07-20T19:01:54.252 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' 2023-07-20T19:01:54.325 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bc77b3c1c5f447c59c9e1d9205273976 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' 2023-07-20T19:01:54.330 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-20T19:01:54.339 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-20T19:01:54.356 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-20T19:01:54.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T19:01:59.373 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T19:01:59.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T19:02:02.910 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-20T19:02:02.921 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-20T19:02:02.922 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T19:02:02.927 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T19:02:02.930 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T19:02:02.947 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2023-07-20T19:02:02.949 INFO:teuthology.orchestra.run.smithi192.stderr:bash: ntpq: command not found 2023-07-20T19:02:02.952 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T19:02:02.952 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2023-07-20T19:02:02.952 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 0 +913us[ +913us] +/- 104ms 2023-07-20T19:02:02.952 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -1966us[-1891us] +/- 78ms 2023-07-20T19:02:02.953 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +1497us[+1497us] +/- 91ms 2023-07-20T19:02:02.953 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T19:02:02.954 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T19:02:02.954 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2023-07-20T19:02:02.954 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 2 +832us[ +832us] +/- 104ms 2023-07-20T19:02:02.954 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -2035us[-1999us] +/- 78ms 2023-07-20T19:02:02.955 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +1425us[+1425us] +/- 91ms 2023-07-20T19:02:02.955 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T19:02:02.968 INFO:teuthology.orchestra.run.smithi057.stderr:bash: ntpq: command not found 2023-07-20T19:02:02.976 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T19:02:02.977 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2023-07-20T19:02:02.977 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +897us[+1000us] +/- 104ms 2023-07-20T19:02:02.978 INFO:teuthology.orchestra.run.smithi057.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -1954us[-1850us] +/- 78ms 2023-07-20T19:02:02.978 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +1515us[+1619us] +/- 91ms 2023-07-20T19:02:02.978 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T19:02:02.979 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-20T19:02:02.988 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-20T19:02:02.989 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-20T19:02:02.997 DEBUG:teuthology.orchestra.run.smithi057:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T19:02:03.198 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T19:02:03.298 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T19:02:03.399 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T19:02:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:02:03.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T19:02:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:02:03.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T19:02:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:02:03.544 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-20T19:02:03.555 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-20T19:02:03.564 INFO:teuthology.task.internal:Duration was 1821.706098 seconds 2023-07-20T19:02:03.564 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-20T19:02:03.573 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-20T19:02:03.573 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T19:02:03.578 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T19:02:03.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T19:02:03.627 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T19:02:03.631 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T19:02:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T19:02:04.106 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-20T19:02:04.107 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2023-07-20T19:02:04.107 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T19:02:04.167 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2023-07-20T19:02:04.168 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T19:02:04.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2023-07-20T19:02:04.193 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T19:02:04.217 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-20T19:02:04.217 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T19:02:04.222 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T19:02:04.234 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T19:02:04.313 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-20T19:02:04.315 DEBUG:teuthology.orchestra.run.smithi057:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T19:02:04.320 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T19:02:04.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T19:02:04.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-20T19:02:04.475 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-20T19:02:04.476 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T19:02:04.546 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T19:02:04.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T19:02:04.601 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-20T19:02:04.611 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:02:04.616 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:02:04.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:02:04.642 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2023-07-20T19:02:04.667 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2023-07-20T19:02:04.680 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2023-07-20T19:02:04.718 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:02:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:02:04.760 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:02:04.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:02:04.777 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:02:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:02:04.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-20T19:02:04.807 INFO:teuthology.task.internal:Transferring archived files... 2023-07-20T19:02:04.808 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/remote/smithi057 2023-07-20T19:02:04.808 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T19:02:04.888 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/remote/smithi088 2023-07-20T19:02:04.889 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T19:02:04.929 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/remote/smithi192 2023-07-20T19:02:04.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T19:02:04.966 INFO:teuthology.task.internal:Removing archive directory... 2023-07-20T19:02:04.967 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T19:02:04.972 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T19:02:04.974 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T19:02:05.018 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-20T19:02:05.038 INFO:teuthology.task.internal:Not uploading archives. 2023-07-20T19:02:05.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-20T19:02:05.050 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-20T19:02:05.050 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T19:02:05.055 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T19:02:05.058 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T19:02:05.071 INFO:teuthology.orchestra.run.smithi088.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 19:02 /home/ubuntu/cephtest 2023-07-20T19:02:05.074 INFO:teuthology.orchestra.run.smithi192.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 19:02 /home/ubuntu/cephtest 2023-07-20T19:02:05.094 INFO:teuthology.orchestra.run.smithi057.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 19:02 /home/ubuntu/cephtest 2023-07-20T19:02:05.097 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-20T19:02:05.107 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-20T19:02:05.127 INFO:teuthology.nuke:Checking targets against current locks 2023-07-20T19:02:05.157 DEBUG:teuthology.nuke:shortname: smithi057 2023-07-20T19:02:05.157 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T19:02:05.183 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:20:24.630419', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T19:02:05.186 DEBUG:teuthology.nuke:shortname: smithi088 2023-07-20T19:02:05.187 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T19:02:05.213 DEBUG:teuthology.nuke:shortname: smithi192 2023-07-20T19:02:05.213 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T19:02:05.216 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:20:24.629175', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T19:02:05.237 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:20:24.627195', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T19:02:05.246 INFO:teuthology.orchestra.console:Power off smithi057 2023-07-20T19:02:05.248 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T19:02:10.306 INFO:teuthology.orchestra.console:Power off smithi192 2023-07-20T19:02:10.307 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T19:02:10.325 INFO:teuthology.orchestra.console:Power off smithi088 2023-07-20T19:02:10.326 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T19:02:10.339 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:02:10.350 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:02:14.362 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:02:14.374 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:02:15.283 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:02:18.474 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:02:18.487 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:02:18.698 INFO:teuthology.orchestra.console:Power off for smithi088 completed 2023-07-20T19:02:20.295 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:02:22.591 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:02:22.805 INFO:teuthology.orchestra.console:Power off for smithi192 completed 2023-07-20T19:02:24.408 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:02:24.619 INFO:teuthology.orchestra.console:Power off for smithi057 completed 2023-07-20T19:02:24.725 INFO:teuthology.run:Summary data: client.0-kernel-sha1: c685da6f56cc77f2cfb9c4221be3960522616caf description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} duration: 1821.7060978412628 failure_reason: 'Command failed on smithi057 with status 2: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bc77b3c1c5f447c59c9e1d9205273976 status: fail success: false 2023-07-20T19:02:24.725 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T19:02:29.811 INFO:teuthology.run:FAIL